sabi-noema-mla

Sabi-Noema-MLA is Sabi-Noema, our agentic model built on top of Qwen3.8-27B, converted to multi-head latent attention (MLA) for a 3.6× smaller attention cache (7.1× with an FP8 cache).

While Sabi-Noema pushes the limits of reasoning efficiency, with Sabi-Noema-MLA we see how much we can compress the memory needed per token so that long agentic sessions fit on consumer GPUs.

⚡ Performance at a Glance

• Attention Cache: 18 KiB per token (vs. 64 KiB base), 3.6× smaller | 9 KiB (7.1× smaller) with an FP8 cache
• Consumer GPUs: 7 concurrent 32K-token sessions on a 24 GB card with 4-bit weights, where the original attention fits 2 (13 with an FP8 cache)
• τ³-bench: 90.6% (vs. 79.1% base)
• Token Efficiency: 1.8× fewer output tokens per Terminal-Bench task (median 36.1K vs. 65.3K)
• Served with a vLLM plugin: sabi-mla-vllm, shipped in this repository's plugin/ directory; also loads in Transformers with trust_remote_code=True

FP8-cache results were measured with a simulated FP8 cache (cached values rounded to FP8 and read back); the released serving path stores a BF16 cache, and real FP8 cache serving is coming.

Resources

• Sabi-Noema — the original-attention model, for maximum accuracy when cache memory is not the constraint

KV Cache Compression

The MLA conversion shrinks the elements per token in the 16 attention layers, and FP8 cache quantization shrinks the bytes per element.

How the Conversion Is Done

In each of the 16 full-attention layers, the per-head key/value cache is replaced by a rank-512 compressed latent plus one 64-dimensional rotary key shared across heads, following DeepSeek-V2[1]. The rotary component is kept separate because position-dependent rotations cannot be absorbed into a low-rank projection; sharing it across heads keeps it to 64 elements per token. The new projections are fit after RL training by matching each layer's attention output to the original Sabi-Noema on long documents of up to 64K tokens. No RL is repeated, so the conversion takes about two hours on a single 8×H100 node.

Memory Requirement

Attention cache for one median-length session on each benchmark (peak prompt-plus-completion per episode, median over episodes). Reductions are against Qwen3.8-27B:

On Terminal-Bench the reduction compounds: 7.1× from the cache format, and 1.18× more because Sabi-Noema-MLA's sessions are shorter (26.0K tokens at the median with the FP8 cache against 30.6K). On the other benchmarks both models use about the same context, so the reduction stays close to the per-token figure. The median understates peak memory on Terminal-Bench: the 90th percentile is 83–95K tokens and the longest session runs to about 200K. Size deployments for the tail, not the median.

Quantizing the MLA cache to FP8 keeps scores at parity: in one paired run, every benchmark is within one task of the BF16 cache, within run-to-run noise.

Sessions per GPU. Cached tokens / concurrent 32K-token sessions:

Assumes 92% of VRAM usable, 1 GiB overhead, weights of about 16.5 / 28 / 55 GB at 4- / 8- / 16-bit, ~72 MiB of linear-attention state per session, and a single sequence capped at the model's 262K context. The 4- and 8-bit weight rows are memory arithmetic only; those weight formats are not evaluated here.

Benchmarks

Sabi-Noema-MLA (BF16 cache) against Qwen3.8-27B and comparison models, with Sabi-Noema for reference. Same harnesses for every model: Terminal-Bench 2.1 following Artificial Analysis's settings on H100 TP2 deployments with up to 8 tasks per server (Sabi-Noema-MLA, whose plugin is single-GPU, and Muse Glimmer on TP1), τ³-bench (airline, retail, telecom) with a GPT-4.1 user simulator, DeepSWE uncapped (seeds with an infrastructure failure scored as a model fail are excluded), ClawEval graded by the harness's LLM judge. Qwen reports 73.0 on Terminal-Bench 2.1 with its own Terminus setup and 42.2 on DeepSWE 1.1 with the Claude Code harness. Terminal-Bench 2.1 is averaged over 2–3 runs per open model and DeepSWE over 2–3 seeds for Sabi-Noema, Sabi-Noema-MLA and Qwen3.8-27B; GPQA-Diamond, IFBench and AA-LCR over 3 runs where we ran them. GPT-6 Luna (openai/gpt-6-luna) was called through OpenRouter at medium reasoning effort; Muse Glimmer