BUD-E-Whisper

Detailed audio captioning model that generates rich, temporal descriptions of speech audio — including voice characteristics, emotional states, recording quality, speaker demographics, and delivery style.

Fine-tuned from laion/BUD-E-Whisper_V1.1 on the majestrino-unified-detailed-captions-temporal dataset.

Model Details

• Architecture: Whisper Small (encoder-decoder, 242M parameters)
• Base model: laion/BUD-E-Whisper_V1.1 (itself fine-tuned from openai/whisper-small)
• Training data: ~9M samples from TTS-AGI/majestrino-unified-detailed-captions-temporal (826 webdataset shards)
• Training: 2x RTX 3090, DDP (gloo), fp16 mixed precision, AdamW (lr=1e-5, linear warmup 5%), batch size 20
• Final validation loss: 0.81

What it outputs

Given an audio clip (up to 30 seconds), the model generates detailed captions describing:

• Speaker demographics: age range, gender, accent
• Voice timbre: pitch, brightness, breathiness, nasality, resonance
• Emotional state: valence, arousal, dominance, specific emotions
• Delivery style: tempo, fluency, expressiveness, naturalness
• Recording quality: background noise, clarity, studio vs. field
• Temporal aspects: how delivery and emotion change over time

Quick Start

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torchaudio
import torch

# Load model
processor = WhisperProcessor.from_pretrained("laion/BUD-E-Whisper_V1.2")
model = WhisperForConditionalGeneration.from_pretrained("laion/BUD-E-Whisper_V1.2")
model.generation_config.forced_decoder_ids = None
model.eval().to("cuda")

# Load audio (resample to 16kHz mono)
wav, sr = torchaudio.load("audio.wav")
if wav.shape[0] > 1:
    wav = wav.mean(dim=0, keepdim=True)
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)
audio = wav.squeeze(0).numpy()

# Generate caption
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt").to("cuda")
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_length=448)
caption = processor.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(caption)

CLI Inference

python inference.py audio.wav
python inference.py audio.mp3 --device cpu

Example Output

Input: A 5-second clip of a male speaker reading a sentence.

Output:

This recording features a clearly masculine adult male speaker delivering a narration with a strong sense of interest and concentration, occasionally exhibiting a hint of contemplation. The overall emotional valence remains neutral, with a slightly calm arousal level throughout. The speaker's demeanor is balanced, displaying a moderate degree of vulnerability alongside a subtle confidence. The vocal delivery is consistently natural and spontaneous, characterized by a neutral pitch and volume, and a smooth, clear timbre with minimal breathiness.

Training Details

Validation Loss Progression

Limitations

• Optimized for speech audio; may produce less meaningful captions for music or environmental sounds
• Maximum input length is 30 seconds
• English-centric training data, though it can handle some other languages (e.g., German)
• May occasionally hallucinate speaker gender or specific emotional states

Citation

@misc{bud-e-whisper-v1.2,
  title={BUD-E-Whisper V1.2: Detailed Audio Captioning},
  author={LAION},
  year={2026},
  url={https://huggingface.co/laion/BUD-E-Whisper_V1.2}
}