Ivme-Conversate

İvme (Turkish: acceleration) is a series of stupidly small language models built to punch above their weight. This is the third release: a 24.79M parameter decoder-only base model trained from scratch, with a deeper/narrower architecture, two new attention-level changes, a digit-atomic tokenizer, and a revised data mix.

v2 fixed v1's biggest problem (coherence) mostly through data — more tokens, better weighted toward staying on topic. v3 keeps that lesson but changes the architecture itself for the first time since v1: more layers, a narrower hidden dimension, and two specific attention modifications (QK-Norm and XSA) chosen because they target coherence directly rather than just adding parameters. The tokenizer was also rebuilt to keep every digit as its own token, aimed squarely at v2's other well-known weak spot: arithmetic.

Model Details

v2 was 10 layers × 384 hidden dim (depth:width ≈ 0.026). v3 deliberately goes deeper and narrower — 16 layers × 320 hidden dim (depth:width ≈ 0.050) — while landing in almost exactly the same parameter budget. Depth tends to help multi-step reasoning and cross-sentence coherence more than raw width does; width tends to help how much can be memorized per token. This is a real rebalancing of the budget, not a token gesture: the embedding table's share of total parameters drops from v2's 25.8% to v3's 20.7%, with the difference going into deeper transformer layers.

QK-Norm: per-head RMSNorm applied to queries and keys before the RoPE rotation and the dot product, for training stability at higher depth and a higher effective learning rate.

XSA (Exclusive Self Attention): after standard scaled-dot-product attention produces output yᵢ for token i, subtract the projection of yᵢ onto that token's own value vector vᵢ:

zᵢ = yᵢ − (yᵢ · vᵢ) / (‖vᵢ‖² + ε) · vᵢ

The idea, from the source paper: standard attention lets a token attend heavily to itself, which can let a model get lazy and just echo its own representation forward instead of pulling in genuinely new context from other tokens. XSA removes that redundant self-component. Zero extra parameters, negligible extra compute.

Benchmarks

Benchmarks were run with lm-evaluation-harness via a transformers-compatible custom model wrapper, plus ArithMark 3.0 via its official evaluation script.

BLiMP moved up a real 3.4 points — the clearest confirmation that the depth/width rebalancing and the two attention changes did what they were meant to do for grammaticality and coherence. WikiText-2 perplexity also improved.

ARC-Easy is flat to slightly down (acc_norm: 39.98% → 39.02%), not up. Worth being straightforward about this rather than spinning it: ARC-Easy tests science-fact recall, which leans more on raw embedding/knowledge capacity than on the kind of cross-sentence reasoning BLiMP measures. v3 deliberately shrank the embedding table's share of the parameter budget (25.8% → 20.7%) to buy depth, and also cut Cosmopedia's weight hard (27% → 5.6%, it was stale synthetic data) in favor of more naturalistic web text. Both are plausible, specific mechanisms for a small knowledge-recall cost alongside the coherence gain — this looks like a real tradeoff from the choices made, not noise.

BLiMP paradigm breakdown

Strong on core agreement paradigms, several near-ceiling:

Weaker on long-distance dependencies and island constraints, same pattern as v2 and typical for small models:

Does the arithmetic tokenizer fix actually work?

Short answer: it fixes the representation, not the reasoning. ArithMark-3 lands at 34.70% (acc_norm), well above the benchmark's 25% random-choice floor and ahead of most other small models in its published leaderboard cohort — a real, positive signal that digit-atomic tokenization helped. But free-form arithmetic generation is still weak: asked directly (1 + 1 =), the model reliably produces plausible-looking numbers that are frequently wrong (1 + 1 = 20).

This split result makes sense once you separate what digit-atomic tokenization actually fixes from what it doesn't. Before v3, multi-digit numbers collapsed into single opaque BPE tokens, so the model never even saw place-value structure — "127" was one indivisible symbol, not 1 hundred + 2 tens + 7 ones. Splitting every digit into its own token fixes that re