Audio8-TTS-Realistic

Languages · Quick start · Serving · Evaluation

Audio8-TTS-Realistic-0.6B builds on Audio8-TTS-Preview-0.6b, delivering substantially improved speech realism while retaining its compact 0.6B architecture and zero-shot voice cloning capabilities.

This release supports 23 languages, adding 12 languages compared with Preview. It uses the same Transformers inference interface as Preview. This repository includes the complete model checkpoint, its 44.1 kHz neural audio codec, tokenizer, processor, and custom Transformers code.

Try speech generation and voice cloning in the interactive Playground. Hosted on Hugging Face ZeroGPU; availability depends on the queue and your daily GPU allowance.

Audio8 is also participating in TTS Arena, where it has received encouraging ratings from listeners. We are grateful to everyone who has listened and voted.

Highlights

• Improved realism: more natural prosody and expressive delivery, building on Preview.
• Broader multilingual coverage: 23 languages, including 12 additions to Preview's 11 supported languages.
• Compact model: 601 million main-model parameters, with the audio codec bundled separately.
• Flexible generation: zero-shot voice cloning from a reference recording and transcript, or speech generation without a reference.
• GPU serving: an SGLang Omni adapter with paged attention, dynamic batching, and an OpenAI-compatible speech API.

Supported Languages

Note: Language quality varies; see the per-language evaluation below. Benchmark coverage includes some additional languages and is distinct from the supported-language list.

Model Details

Audio8 TTS uses a DualAR architecture inspired by Fish Audio S2 Pro. The slow AR transformer predicts one semantic token for each audio frame. The fast AR transformer predicts the frame's codec codebooks, conditioned on the slow hidden state and preceding codebooks.

The bundled codec handles both reference-audio encoding and waveform decoding, so no additional codec checkpoint is required. The 0.6B size refers to the main DualAR model; the codec is excluded from that parameter count.

Installation

Python 3.10 or newer and a CUDA-capable GPU are recommended.

pip install "torch>=2.5.0" "torchaudio>=2.5.0" \
  "transformers>=4.57.0,=0.12" "safetensors>=0.4"

The Transformers release package was smoke-tested with PyTorch 2.8.0 and Transformers 4.57.3 on an NVIDIA H20 using BF16. Checks covered model loading, English and Chinese generation with and without reference audio, and generation in Arabic, Cantonese, Portuguese, and Russian. These are runtime checks, separate from the benchmark results below.

Usage

The model uses custom Transformers code. Review the files in this repository, then load it with trust_remote_code=True.

Zero-shot voice cloning

Use a clean reference recording. Its transcript must match the spoken content exactly.

import soundfile as sf
import torch
from transformers import AutoModel, AutoProcessor

model_id = "Edge0/Audio8-TTS-Realistic-0.6b"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).eval().to(device)

inputs = processor(
    text=["Welcome to Audio8 TTS. It is wonderful to hear from you."],
    reference_audio=["reference.wav"],
    reference_text=["The exact transcript of the reference recording."],
    return_tensors="pt",
)
inputs = {name: value.to(device) for name, value in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        do_sample=True,
        return_dict_in_generate=True,
    )
    waveforms, waveform_lengths = model.decode_audio(output.codes)

audio = waveforms[0, : int(waveform_lengths[0])].float().cpu().numpy()
sf.write("output.wav", audio, model.config.codec_sample_rate)

Generation without a reference

Replace the processor call above with:

inputs = processor(
    text=["This utterance does not use a reference voice."],
    return_tensors="pt",
)

Then move the inputs to the model device and run the same generation and decoding steps. For command-line inference, batching, and supervised fine-tuning, see the Audio8 TTS repository, using Edge0/Audio8-TTS-Realistic-0.6b as the model path.

High-throughput serving: SGLang Omni

The [SGLang Omni adapter](https://github.com/Edge0-AI/Audio8_TTS/tre