Model reference · open weights
TinyLlama-intermediate-step-1431k-3T is an open-weight language model from TinyLlama. TinyLlama-1.1B-intermediate-step-1431k-3T (FP32) weighs 2.2 GB; the smallest configuration that runs it is RTX 3060 12 GB.
For self-hosting, compare this build's requirements below with the GPU you plan to rent.
Summary of the TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T model card, 2026-10-01
What it is
| Released by | TinyLlama |
|---|---|
| Released | 2023-12-28 |
| Parameters | 1.1B |
| VRAM | 2.2 GB for the weights |
What it runs on
| Card | Requests at once | Context max | Memory |
|---|---|---|---|
| 2K each | |||
| RTX 3060 12 GB | 192 | all 2K | 11.6 GB |
| RTX 4060 Ti 16 GB | 274 | all 2K | 15.4 GB |
| RTX 3090 24 GB | 447 | all 2K | 23.4 GB |
| RTX 4090 24 GB | 446 | all 2K | 23.4 GB |
| RTX 5090 32 GB | 612 | all 2K | 31.0 GB |
| L40S 48 GB | 893 | all 2K | 44.0 GB |
| A100 80 GB | 1000+ | all 2K | 78.2 GB |
| H100 80 GB | 1000+ | all 2K | 78.1 GB |
| RTX PRO 6000 Blackwell 96 GB | 1000+ | all 2K | 93.8 GB |
| DGX Spark (GB10) 128 GB unified | 1000+ | all 2K | 107 GB |
| H200 141 GB | 1000+ | all 2K | 138 GB |
| B200 180 GB | 1000+ | all 2K | 176 GB |
| Requests at once | 2K tokens each |
|---|---|
| 1 | 2.8 GB |
| 5 | 3.0 GB |
| 8 | 3.1 GB |
| 16 | 3.5 GB |
| 32 | 4.2 GB |
| 64 | 5.7 GB |
One card, with vLLM's small-card settings.
From the model card
https://github.com/jzhang38/TinyLlama
The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01.
We adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint.
This collection contains all checkpoints after the 1T fix. Branch name indicates the step and number of tokens seen.
| Model | Pretrain Tokens | HellaSwag | Obqa | WinoGrande | ARC_c | ARC_e | boolq | piqa | avg |
|---|---|---|---|---|---|---|---|---|---|
| Pythia-1.0B | 300B | 47.16 | 31.40 | 53.43 | 27.05 | 48.99 | 60.83 | 69.21 | 48.30 |
| TinyLlama-1.1B-intermediate-step-50K-104b | 103B | 43.50 | 29.80 | 53.28 | 24.32 | 44.91 | 59.66 | 67.30 | 46.11 |
| TinyLlama-1.1B-intermediate-step-240k-503b | 503B | 49.56 | 31.40 | 55.80 | 26.54 | 48.32 | 56.91 | 69.42 | 48.28 |
| TinyLlama-1.1B-intermediate-step-480k-1007B | 1007B | 52.54 | 33.40 | 55.96 | 27.82 | 52.36 | 59.54 | 69.91 | 50.22 |
| TinyLlama-1.1B-intermediate-step-715k-1.5T | 1.5T | 53.68 | 35.20 | 58.33 | 29.18 | 51.89 | 59.08 | 71.65 | 51.29 |
| TinyLlama-1.1B-intermediate-step-955k-2T | 2T | 54.63 | 33.40 | 56.83 | 28.07 | 54.67 | 63.21 | 70.67 | 51.64 |
| TinyLlama-1.1B-intermediate-step-1195k-2.5T | 2.5T | 58.96 | 34.40 | 58.72 | 31.91 | 56.78 | 63.21 | 73.07 | 53.86 |
| TinyLlama-1.1B-intermediate-step-1431k-3T | 3T | 59.20 | 36.00 | 59.12 | 30.12 | 55.25 | 57.83 | 73.29 | 52.99 |
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 36.42 |
| AI2 Reasoning Challenge (25-Shot) | 33.87 |
| HellaSwag (10-Shot) | 60.31 |
| MMLU (5-Shot) | 26.04 |
| TruthfulQA (0-shot) | 37.32 |
| Winogrande (5-shot) | 59.51 |
| GSM8k (5-shot) | 1.44 |
Quoted from the model card on Hugging Face. The full card is behind the Hugging Face link above.
Benchmarks
As published on the model card: the maker's own numbers, not measured by AxForge.
| Task | Dataset | Metric | Score |
|---|---|---|---|
| Text Generation | AI2 Reasoning Challenge (25-Shot) | normalized accuracy | 33.870 |
| Text Generation | HellaSwag (10-Shot) | normalized accuracy | 60.310 |
| Text Generation | MMLU (5-Shot) | accuracy | 26.040 |
| Text Generation | TruthfulQA (0-shot) | mc2 | 37.320 |
| Text Generation | Winogrande (5-shot) | accuracy | 59.510 |
| Text Generation | GSM8k (5-shot) | accuracy | 1.440 |