RubatoCodec

RubatoCodec is a neural speech codec that divides speech into a fixed number of variable-duration events. Each event carries discrete code indices and an explicit duration. Event placement adapts to speech content, while the number of events is determined exactly by the utterance length, so the token sequence length a speech generator must produce is predictable. A causal decoder renders completed events incrementally, which makes the codec suitable for streaming text-to-speech.

The name comes from the musical term rubato: local timing flexes while the overall tempo is preserved.

• Adaptive Firing with an exact event budget. Multi-scale latent dissimilarity and waveform spectral flux guide cumulative allocation on a 50-Hz latent grid. For T frames the event count is exactly max(1, min(T, round(0.25 · T))), giving 12.5 events/s. No learned boundary predictor, boundary labels, or rate loss is used.
• Streaming decoding. Events can be decoded one at a time or in chunks. Event-by-event decoding takes about 11 ms per event on an NVIDIA H200, well below the ~80 ms of audio each event represents on average.
• Hybrid residual quantizer with depth control. A semantically supervised FSQ first layer is followed by 15 DAC-style residual VQ layers, each with 4,096 entries. Any prefix depth from 1 to 16 can be decoded, giving code-index bitrates of 0.15–2.4 kbit/s plus duration information.
• Multilingual. Trained on approximately 375,000 hours of speech, mostly English, Japanese, and Chinese, with 16-kHz input and 44.1-kHz output.

Model details

Usage

Install the package from the GitHub repository:

git clone https://github.com/third-intelligence/RubatoCodec
cd RubatoCodec
uv sync

Encode and decode:

import soundfile as sf
import torch

from rubatocodec import RubatoCodec

codec = RubatoCodec.from_pretrained("third-intelligence/RubatoCodec", device="cuda")

data, sr = sf.read("speech.wav", dtype="float32")  # mono waveform
wav = torch.from_numpy(data)

enc = codec.encode(wav, sr=sr)       # {"codes": (16, K) int64, "durs": (K,) int64}
audio = codec.decode(enc)            # (1, 1, T_audio) at 44.1 kHz, full depth
audio = codec.decode(enc, depth=8)   # lower bitrate with the first 8 layers

sf.write("recon.wav", audio[0, 0].cpu().numpy(), codec.output_sr)

durs gives each event's duration in 20-ms grid frames. Loading also downloads facebook/w2v-bert-2.0, which is used only for encoding.

Streaming decoding, inference settings, the code format, and training are documented in the repository: inference guide, architecture, training.

Evaluation

RubatoCodec is evaluated on LibriSpeech test-clean, VCTK, JVS nonpara30, JVNV, and seed-tts-eval test-zh against 13 other codecs. At comparable bitrates, it shows the best intelligibility preservation among the compared codecs on the English and Japanese benchmarks and high predicted quality, while the strongest comparators achieve higher PESQ-WB and STOI.

dWER/dCER compare ASR transcripts of the reconstructed and original audio (Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is cosine similarity of speaker embeddings from a fine-tuned WavLM-Large speaker verification model. Bitrates include duration information.

The tables below show a subset of the compared codecs. Bold and band.

LibriSpeech test-clean (English)

JVNV (Japanese emotional speech with nonverbal vocalizations)

On expressive speech, RubatoCodec keeps dCER below 2% at both bitrates, while every other compared codec exceeds 3%.