tinctura

A 96.2M parameter decoder-only language model pretrained from scratch on 75B tokens. It is the cagliostro-v3 recipe at two thirds of the size: the same architecture, data, schedule and token count, with 18 layers instead of 30.

On the Open SLM Leaderboard Index it scores 20.81. Among models under 100M parameters that places it second, 0.26 behind Rose-1.5-Medium at 21.07.

The training run is complete. 75.00B tokens, 762,939 steps, learning rate decayed to zero.

Results

Zero-shot, measured on the public weights in this repository with exactly the commands under Reproducing the evaluation: the lm-evaluation-harness hf backend and the leaderboard's official ArithMark-3 script, both in float32.

The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge. Everything further down that tracks training uses the training-time evaluation harness. That harness tokenizes each answer separately from its question and ran ArithMark-3 in bfloat16, so it reads a little differently: 20.77 for the final checkpoint, with individual benchmarks up to 0.7 points apart from the table above. The numbers in this table are the ones to compare against the leaderboard.

The top of the sub-100M field, using the leaderboard's published figures. tinctura-v1 is scored with the same formula but is not yet listed on the board.

Against the leader the result is split. tinctura-v1 is ahead on ARC-Easy and PIQA, level on HellaSwag, and behind on ARC-Challenge and ArithMark-3. ArithMark-3 is most of the gap: 2.3 points there cost about 0.55 Index at its 0.65 weight, more than the whole 0.26 margin. Rose-1.5-Medium was trained on roughly 100B tokens according to its card, against 75B here.

What a third fewer parameters costs

Because tinctura-v1 follows cagliostro-v3 step for step, the two runs can be compared at identical points in training. Several of these comparisons were run side by side on the same machine with the same evaluation code, using checkpoints from both repositories. Every number in this section comes from the training-time harness, for both models.

Through the stable phase the smaller model trailed by about 1.7 Index over the first 20B tokens and by about 2.9 after that. Single evaluations carry roughly one point of noise, so individual gaps ranged from 0.5 to 3.9. Just before the cooldown, at step 646,000, the paired evaluation read 19.34 against 21.96.

The cooldown is where the two models part. cagliostro-v3 gained 4.6 Index over the last 15% of training. tinctura-v1 gained 1.4. The shortfall came in two places. From step 646,000 to 706,000 cagliostro-v3 added 1.95 and tinctura-v1 only 0.21. From 706,000 to 736,000 the two moved together, 1.32 and 1.23. Over the final stretch to the end cagliostro-v3 added another 1.32 while tinctura-v1 held flat. On ARC-Challenge and ArithMark-3 the smaller model finished the cooldown slightly lower than it started it.

After the first few billion tokens the training loss gap to cagliostro-v3 settled between 0.07 and 0.10 and stayed there through the cooldown. The sharp drop at 63.75B is the data mixture changing, not the model improving.

The cooldown was run twice

The first attempt at the cooldown hit a bug in the data loader. When training switches to the cooldown mixture, each process restarts its data-loading threads. Threads blocked on a full queue could miss the stop signal and keep producing batches from the old mixture next to the new ones, so each process trained on a blend. A standalone test of the loader leaked old-mixture batches in 9 of 10 trials, 20 to 44% of batches after the switch. The loss told on it: at the switch the loss fell by 0.26 where cagliostro-v3's fell by 0.49.

The loader was fixed so that every restart gets its own queue and old threads exit on their own, and the fixed version leaked nothing in the same test. The cooldown was then rerun from the last checkpoint before it, step 644,000. The rerun's loss fell by 0.45 at the switch and tracked cagliostro-v3 as expected. The weights in this repository come from the rerun.

The fix did not change the benchmarks in a measurable way. At step 706,000 the first attempt scored 19.68 and the rerun 19.55, which is within evaluation noise. The data-mixing bug was real, but it was not what held the cooldown gain down.

Model details

The architecture is defined in this repository and is the same code as cagliostro-v3. trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers.

Training data

The same mixtures and shards as cagliostro-v3. The first mixture covers the stable phase, and the second takes over when the cooldown begins at 85% of