moss-tts-local-transformer-voice-acting

An expressive voice-acting model: fantasy characters (orc / dragon / fairy / goblin), shouting, whispering, and vocal bursts (laughs, gasps, sighs, giggles, moans) driven by natural-language director instructions, in English and German.

Authors: Christoph Schuhmann and LAION.

What is new in v2

This checkpoint is the latest and best in the line. It was obtained by taking the previous release — laion/moss-tts-local-transformer-4.55b-voice-acting — and running further supervised fine-tuning on additional synthetic voice-acting data. The extra data was self-generated with the model family and filtered with a Best-of-N reward (a lightweight VoiceCLAP-based genuineness + vocal-burst-blend + character reward that keeps only the strongest of many candidate takes per prompt), so the model learns from its own highest-quality performances — more consistent character delivery, more natural vocal bursts, and cleaner emotional range. It remains a full 4.55-billion-parameter local-transformer MOSS variant at 48 kHz and is a drop-in replacement for the previous release (same architecture, tokenizer, and prompt format).

Its sibling is the 8-billion-parameter delay-architecture model at 24 kHz: laion/moss-tts-v1.5-8b-voice-acting.

Off-the-shelf: this repository contains the fully merged weights (bf16). Load and generate; no adapter handling needed.

Further fine-tuning (v2)

• Additional data: a large corpus of synthetically generated, Best-of-N-filtered voice-acting performances, self-distilled from the model family — structured natural-language director instructions paired with spoken lines, covering fantasy characters, strong emotions, and vocal bursts (laughs, gasps, sighs, moans) in English and German. No third-party dataset names are disclosed.
• Objective / filtering: candidates were ranked by a cheap automatic reward (VoiceCLAP-commercial embedding → genuineness + vocal-burst-blend + character probes) and only the top takes were kept, so the model is trained on its own strongest performances rather than raw generations.
• Setup: 8×GPU DDP, fp32 master weights + 8-bit optimizer with bf16 compute (avoids the update-stagnation that pure-bf16 parameter fine-tuning suffers), gradient checkpointing, per-sample caption augmentation (randomly dropping the general voice description, the inline stage directions, or a fraction of the inline cues/pauses so the model learns to improvise them) and reference-audio dropout, linear LR schedule with warmup, channel-wise RVQ loss weighting. Released weights are the best checkpoint by cumulative genuineness + vocal-burst-blend on a held-out validation set.

Requirements

pip install "transformers>=4.45" torch torchaudio safetensors huggingface_hub

The MOSS v2 audio codec (OpenMOSS-Team/MOSS-Audio-Tokenizer-v2) is pulled automatically. ~10 GB VRAM (bf16 + codec).

Prompt format — GENERAL: / SCRIPT: + plain text

The model was trained on two fields per example:

• instruction = the performance direction, written as two labelled blocks:
  • GENERAL: — the overall voice: age / gender / timbre / mood / delivery. For the most natural result, also state that it is a pristine, high-quality studio recording with no background noise and a genuine, spontaneous delivery ("like a real person in a real moment, not acted").
  • SCRIPT: — the spoken lines again, annotated in position with (delivery cues), vocal bursts such as (a soft laugh) / (gasp) / (sigh), and [pause] markers. The emotional arc can change line-by-line here — this is where you direct the performance.
• text = just the plain spoken words — no cues, no brackets.

The words appear in both fields (plain in text, cue-annotated inside the SCRIPT block) — this is intentional and matches the training data. These two strings map directly onto generate(text=..., instruction=...) in the inference example below.

Exact template

instruction =
GENERAL:
SCRIPT:
(cue, cue, ...) first line of dialogue [pause] rest of the line
(cue, cue, ...) second line of dialogue
...

text =

Example (an emotional arc: warm → shocked → furious)

instruction =
GENERAL: A young man's voice — bright, warm and overjoyed, then it curdles into shocked disbelief and finally erupts into loud, aggressive, disgusted ranting. Completely natural and spontaneous, like a real person in a real moment, not acted. Pristine high-quality studio recording, no background noise.
SCRIPT:
(lighting up, warm, breathless with delight) Oh my god— hey! [pause] Is that really you?
(beaming, affectionate, a soft happy laugh) I can't believe it, look at you!
(the smile fading, confused, voice tightening) ...Wait. What is that?
(erupting, very loud, aggressive, furious and betrayed) You actually did it?! After everything I did for you?!
(ranting fast, seething, disgusted) God, it makes me sick. Get out of my sight.

text =
Oh m