glm-4-voice-of-reason-stitch

A speech-to-speech model that reasons while it speaks. Starting from GLM-4-Voice-9B, it was supervised-finetuned on stitched dialogues — unspoken written reasoning chunks alternating with spoken response chunks — then trained with reinforcement learning against a binary LLM judge on math word problems.

The interleaving follows STITCH (arXiv:2507.15375, Chiang et al., ICLR 2026): a chunk of speech takes far longer to play than to generate, and the reasoning tokens are emitted in that spare time, so thinking costs no extra latency.

Scores 0.771 on GSM8K (1310 test items, written channel, gpt-4o-2024-11-20 as judge).

Run it on a wav

The audio front end is GLM-4-Voice's, unchanged and not in this repo, so clone it for the speech tokenizer:

git clone https://github.com/THUDM/GLM-4-Voice
uv init glm-of-reason && cd glm-of-reason
uv add "transformers>=4.44,<4.48" torch torchaudio accelerate tiktoken soundfile

tiktoken is needed by the tokenizer's remote code and soundfile by torchaudio to read your file; both fail late and unhelpfully if missing. Save the script below as demo.py, point AUDIO at your wav — anything torchaudio can read, any sample rate — and run uv run demo.py:

import re, sys, torch
sys.path.insert(0, "../GLM-4-Voice")           # the clone, next to the project

from transformers import AutoModel, AutoTokenizer, WhisperFeatureExtractor
from speech_tokenizer.modeling_whisper import WhisperVQEncoder
from speech_tokenizer.utils import extract_speech_token

AUDIO = "question.wav"                                    # <-- your spoken question
REPO = "kyutai/glm-4-voice-of-reason-stitch-9b"

SYSTEM = (
    "User will provide you with a speech instruction. Do it step by step. "
    "First think in partial reasoning chunks of 100 tokens using [SOPR] and [EOPR], "
    "then respond in an interleaved manner, with 13 text tokens followed by 26 audio tokens."
)

tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
model = AutoModel.from_pretrained(
    REPO, torch_dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
).eval()

whisper = WhisperVQEncoder.from_pretrained("THUDM/glm-4-voice-tokenizer").eval().to("cuda")
features = WhisperFeatureExtractor.from_pretrained("THUDM/glm-4-voice-tokenizer")

audio_tokens = extract_speech_token(whisper, features, [AUDIO])[0]
user = "" + "".join(f"" for t in audio_tokens) + ""
prompt = f"\n{SYSTEM}\n{user}streaming_transcription\n"

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **inputs,
    max_new_tokens=1000,
    do_sample=False,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.convert_tokens_to_ids(""),
)
new = out[0, inputs.input_ids.shape[1]:].tolist()

audio_offset = tokenizer.convert_tokens_to_ids("")
text = tokenizer.decode([t for t in new if t < audio_offset], skip_special_tokens=True)
speech_tokens = [t - audio_offset for t in new if t >= audio_offset]

print(text)                                               # reasoning and answer, interleaved
print(re.sub(r"\[SOPR\].*?(\[EOPR\]|$)", "", text, flags=re.S))   # only what is spoken

text comes out with the unspoken reasoning wrapped in [SOPR] ... [EOPR]; everything outside those markers is the spoken answer, which is why the second print is the one to read aloud. For example:

[SOPR]A navarin is a traditional dish ... the type of meat used in a navarin is lamb. The answer
is[EOPR]A navarin is a Middle Eastern dish made with lamb, so[SOPR] lamb.[EOPR] the meat used is lamb.

speech_tokens are the audio codes of that same answer; feed them to THUDM/glm-4-voice-decoder to get a waveform, or run the full duplex demo from the GLM-4-Voice repo with --model-path pointing here.

This script was run as printed, on one H100, with transformers 4.47.1 and torch 2.8.0. Pick a torch build that matches your driver: the newest wheel needs a newer CUDA than many clusters run.

License

Inherited from GLM-4-Voice; see the license link above.