GPT-X3

GPT-X3-150M is Axiomic Labs’ third-generation base model, built on the TX5 hybrid GDN2/gated GQA architecture and trained from scratch with TrainWork. It ranks #1 on the Open SLM Leaderboard and is the first model under 150M parameters to exceed an Intelligence Index of 30.

Performance

Data Efficiency

GPT-X3-150M outperforms Hugging Face's SmolLM2-135M on the Intelligence Index while using about 27× fewer training tokens (75B vs 2T). Benchmark results and Intelligence Index methodology follow the Open SLM Leaderboard. Training compute is estimated as 6 × parameters × training tokens.

Relative Leaderboard Performance

The Intelligence Index chance-normalizes HellaSwag, combined ARC, PIQA, and ArithMark-3, then applies weights of 1.00, 1.00, 1.00, and 0.65 respectively.

What's New in X3 vs GPT-X2.5?

Model size and architecture

• 30 XGQA layers -> 24 hybrid layers (18 GDN2 + 6 gated GQA).
  • Three GDN2 layers between each full-attention layer make the stack a 3:1 hybrid.
• XSA -> normalized, gated GQA with RoPE on 32/64 head dimensions.
  • Adds query/key normalization and an output gate to X3's full-attention layers. Ablations showed XSA redundant with attention gating.
• 32,770 -> 65,536 vocabulary tokens; uses the LFM2.5 tokenizer.
  • Larger tokenizer yields across the board improvements including speed, training efficiency, long context performance, etc.

Training and optimization

• 11-source progressive mix -> nine-source fixed mix.
  • X3 keeps the source proportions constant throughout training as small models struggle to adapt to data changes.
• Next-token training -> six-token TST for the first 30%, then next-token training; 75B compute tokens and ~187.5B raw tokens total.
  • TST predicts six target tokens per compute position during its phase.
• AdamW -> hybrid Muon + AdamW.
  • Muon handles matrix weights; AdamW handles embeddings, norms, biases, and other non-matrix parameters.

Architecture

Config

vocab_size             = 65,536    (LFM2.5 tokenizer no BOS)
n_layer                = 24
n_embd                 = 576
block_size             = 2048
gdn_head_dim           = 64        (9 GDN2 heads)
gqa_head_dim           = 64        (9 query heads)
gqa_n_kv_heads         = 3         (3:1 grouped-query attention)
gqa_rotary_dim         = 32
gdn_layers_per_gqa     = 3         (GDN2:GQA = 3:1)
swiglu_hidden_ratio    = 2.75      (1,584 units)
rope_theta             = 100,000

Parameter Breakdown

Training

GPT-X3 is trained for 143,052 steps across two GPUs, with a global batch of 524,288 compute tokens per step. The run uses six-token TST for the first 30% of training, then switches to standard next-token prediction.

Data

The configured source folders are mixed from step 0 with one fixed set of weights; there are no later curriculum stages. The run uses 143,052 steps at a 524,288-token global batch (about 75.0004B compute tokens). Token-Superposition Training (TST) is enabled for the first 30% of the run with six-token bags, consuming about 135B raw tokens in that phase before standard autoregressive training resumes.