KLM-1.5

KLM 1.5 is a 99M-parameter base language model. It is a new pretrain compared to KLM 1.1 which continued training KeyLM-75M whereas KLM 1.5 has a new architecture and was trained from scratch on 100B tokens.

What's new

• Hybrid layers. KLM 1.5 mixes Gated DeltaNet-2 and full attention layers at a 3:1 ratio. The Gated DeltaNet-2 layers carry a fixed-size recurrent memory through the sequence instead of attending to every earlier token.
• Block Attention Residuals (Kimi Team, 2026). Each layer reads a learned mix of the embeddings and the outputs of every earlier layer, instead of adding everything into one running residual sum.
• Exact n-gram memory. The 125,000 most frequent bigrams and 38,000 most frequent trigrams in the training data each have their own learned vector, which is looked up exactly and fed into the model next to the token embeddings. The memory holds 5.4M parameters.
• A small tokenizer. The vocabulary has 4,100 tokens and splits every number into single digits. Since the embeddings are tied, they only cost 2.1M parameters, which leaves almost all of the model for the layers.

Benchmarks

All scores are zero-shot with length-normalized accuracy.

• ARC, HellaSwag and PIQA were scored using lm-eval 0.4.12.
• ArithMark-3 was scored using the official ArithMark-3.0 script.
• Peer scores are taken from the Open SLM leaderboard as of 2026-10-22.

On that leaderboard, KLM 1.5's Intelligence Index is 30.26 which puts it first among models under 100M parameters and second under 150M.

Each chart's axis starts at random-guess accuracy, which is 25% for most benchmarks and 50% for PIQA, so the height of a bar shows how far a model scores above chance.

Training

KLM 1.5 was trained on 2,048-token sequences with 128 sequences per step, in three stages.

1. Superposition warm-up for 70,000 steps, using Token-Superposition Training. Each input position is the average of 7 consecutive tokens, and the model learns to predict the next 7 tokens as an unordered group. This lets the warm-up cover 128B tokens of ClimbMix for the compute of about 18B.
2. Stable training on 80B tokens of next-token prediction at a constant learning rate.
3. Decay on 20B tokens of a mix with more math and Q&A, while the learning rate falls to zero along a 1−√t curve.

The weight matrices were trained with MuonEq-R at a learning rate of 0.01, and the embeddings, norms and gates with AdamW at 3e-3, in bf16. Each 2,048-token sequence packs several documents, and the model never looks across a document boundary since attention is masked there, and the recurrent state and n-gram history are reset.

The data mix shifted between the stable and decay stages as follows.

Architecture

Intended use

KLM 1.5 is a base model. It continues text but has not been trained to follow instructions or chat, so it is meant to be fine-tuned for your own use case. The tokenizer already has , , and as single tokens, so they are ready to use in chat and reasoning fine-tunes.

Usage

The model code is plain PyTorch, so KLM 1.5 runs on a CPU as well as on any GPU without extra packages beyond Transformers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MinimaLabs/KLM-1.5"
device = "cuda" if torc