dots.tts-soar

dots.tts is a 2B-parameter fully continuous, end-to-end autoregressive (AR) text-to-speech system. The backbone pairs a semantic encoder, an LLM, and an autoregressive flow-matching acoustic head over a 48 kHz AudioVAE — no discrete codec tokens anywhere in the pipeline.

This repository hosts dots.tts-soar — the pretrained backbone further refined with Self-corrective Alignment (SCA), a reward-free flow-matching-native post-training stage. SCA pushes the model to the highest zero-shot fidelity and speaker similarity of the three releases and is the recommended default for production zero-shot voice cloning.

Quick Start

Installation

conda create -n dots_tts python=3.10 -y
conda activate dots_tts

python -m pip install --upgrade pip
python -m pip install "git+https://github.com/studio-dots-ai/dots.tts.git" \
  -c "https://raw.githubusercontent.com/studio-dots-ai/dots.tts/main/constraints/recommended.txt"

CLI

# Continuation voice cloning (reference audio + transcript) — recommended
dots.tts \
  --model-name-or-path dots-studio/dots.tts-soar \
  --text "Hello, this is a zero-shot voice cloning demonstration." \
  --prompt-audio /path/to/reference.wav \
  --prompt-text "The exact transcript of the reference audio." \
  --output clone.wav

Python API

from dots_tts.runtime import DotsTtsRuntime
import soundfile as sf

runtime = DotsTtsRuntime.from_pretrained(
    "dots-studio/dots.tts-soar",
    precision="bfloat16",
)

result = runtime.generate(
    text="Hello, this is a quick speech synthesis test.",
    prompt_audio_path="/path/to/reference.wav",
    prompt_text="The exact transcript of the reference audio.",
    num_steps=10,
    guidance_scale=1.2,
)

sf.write("output.wav", result["audio"].float().cpu().squeeze().numpy(), result["sample_rate"])

Recommended sampling settings

Fine-tuning

Both dots.tts-base and dots.tts-soar are valid fine-tuning starting points. Pick dots.tts-soar when you want to inherit its tightened text/timbre alignment on top of the pretrained backbone. See the training script and smoke config in the source repository:

accelerate launch scripts/train_dots_tts.py --config configs/dots_tts.yaml

Architecture

A frozen AudioVAE encodes 48 kHz mono waveform into a continuous latent and decodes it back via a BigVGAN-style causal decoder. An autoregressive backbone predicts that latent one patch at a time:

• Semantic encoder — re-encodes each newly generated VAE patch into a compact embedding for the LLM, stripping high-variance acoustic detail.
• LLM — initialized from Qwen2.5-1.5B-Base, consumes BPE text directly (no phonemes), emits one hidden state per audio step.
• AR flow-matching head — a DiT that conditions on the LLM hidden state and the AR prefix to denoise the next VAE patch, with a frozen CAM++ speaker x-vector as side input.

Self-corrective Alignment is a reward-free, flow-matching-native post-training stage applied on top of dots.tts-base. It improves text and speaker adherence without changing inference cost or sampling schedule.

Performance — dots.tts-soar

Seed-TTS-Eval — state-of-the-art average SIM (79.2)

MiniMax Multilingual — highest average SIM (83.9) across 24 languages

CV3-Eval — leads both cross-lingual SIM subsets

EmergentTTS-Eval — top Syntactic Complexity in the table (65.7%)

On head-to-head judging vs. gpt-4o-mini-tts, dots.tts-soar posts 65.7% on Syntactic Complexity — above every closed-source system listed, while keeping competitive Emotions / Questions scores.

See the project README for full benchmark tables.

Risks and Limitations

• Misuse risk. High-fidelity zero-shot voice cloning can produce highly realistic synthetic speech. This checkpoint is intended for research and authorized deployment. Do not use it for impersonation, fraud, or disinformation. Combine downstream use with consent-aware reference-audio policies, robust synthetic-speech detection, and content watermarking. Clearly mark AI-generated audio.
• Low-resource WER gap. A BPE backbone inherits the text LLM's language coverage at the cost of a higher data appetite. On script-divergent and under-represented languages (Arabic, Hindi, Turkish, Vietnamese) WER is higher than on high-resource languages; speaker similarity is preserved.
• Speech-heavy training. The backbone is trained on a speech-heavy mixture. Singing and unified speech + so