SupraNeo

A 4.07M-parameter decoder-only language model built to test a single question: how much of a tiny model should actually compute anything?

Most models at this scale spend the majority of their parameters on a vocabulary lookup table. SupraNeo-4M spends 82% on the transformer stack. The design starts from a custom 4,096-token BPE tokenizer, which drops the embedding matrix to 524k parameters and frees the rest for a deep, narrow stack. The aspect ratio (d/L ≈ 13) is deliberately below what the 100M class uses — following the MobileLLM finding that optimal depth-to-width shifts toward depth as models shrink.

Architecture

No custom modeling code — trust_remote_code is not required.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/SupraNeo-4M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/SupraNeo-4M")

ids = tok("The main reason that", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=40, temperature=0.8, top_p=0.9, do_sample=True)
print(tok.decode(out[0]))

This is a base model with no chat template. apply_chat_template will fail by design.

Training

Pre-trained from scratch on a single NVIDIA L4 over 5B tokens (~1,230 tokens per parameter).

The heavy FineWeb-Edu weighting is deliberate. DCLM-baseline was tuned to win reasoning benchmarks in the 1–7B range, and none of that transfers at 4M — what a model this size can learn is register and local fluency, and FineWeb-Edu's uniform expository prose is far easier to model.

Evaluation — BananaMind Base Bench 1.1

350 items, 4-way continuation-likelihood, add_special_tokens=False, no BOS, selection by highest conditional mean log-prob, Elo by weighted MLE with a prior of 4 games at 1000. Official run: dataset checksum verified, schema verified, 0 truncated contexts, 0 truncated continuations.

By difficulty: easy 39.3%, medium 29.1%, hard 31.9%.

Reading these numbers honestly

The aggregate is significantly above chance, but the signal is concentrated in one place. Language Completion at 56% (+5.06σ) is the only strongly separated category, and that is exactly what a 4M model should be able to do: local grammatical and register plausibility. Logical Reasoning clears the bar marginally. The remaining five categories sit within noise, and Quantitative and Code Completion land at or slightly below chance — this model has no arithmetic or code capability, and the card should not be read as claiming otherwise.

The medium/hard inversion (29.1% vs 31.9%) is noise at n≈117, not evidence that harder items are easier.

BananaMind Base Bench was calibrated for the 65M–100M+ range. At 4M, with 350 four-way items, the detection floor at 1.96σ is roughly 29.5% accuracy — most of this benchmark simply lacks resolution here. For tracking progress at this scale, bits-per-byte on held-out text and BLiMP are the metrics with actual sensitivity. HellaSwag, PIQA and ARC are not reported because they sit at chance and measure nothing.

Limitations

SupraNeo-4M produces grammatical, register-consistent English with coherence over one to two sentences and topical drift beyond that. It has no factual reliability, no arithmetic, no code ability, and no instruction following. The 4,096-token vocabulary means it compresses text ~35% less efficiently than a standard 32k tokenizer, and its outputs are not comparable to other models by raw cross-entropy — use bits-per-byte.

This is a research artifact for studying small-scale pretraining, vocabulary budgets, and data mixtures. It is not intended for deployment.

Carbon footprint

Training was estimated to emit 0.46 kg CO₂ eq. — a single L4 for the duration of the run. For reference, that is roughly the footprint of driving a passenger car about 2.5 kilometers.

by DedeProGames