NCP_ArchPreview_dolma3_8.9B_Stage2

Model collection | Technical report (arXiv) | HF Papers | Training code (coming soon) | Evaluation code

NCP-ArchPreview Stage 2 continues the Stage 1 model on Dolma 3 Dolmino, following the second stage of the OLMo-3 data curriculum. It retains joint Next Token Prediction (NTP) and Next Concept Prediction (NCP), allowing the model to refine token-level capabilities and its learned concept space during continued pretraining.

Developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University, the model has approximately 8.94B parameters and an 8,192-token training context. Stage 2 remains a base-model release for completion, evaluation, and further adaptation.

Highlights

• Continued training in token and concept space. Stage 2 preserves the 16-layer Encoder, 8-layer Concept Module, and 16-layer Decoder from Stage 1.
• Improved main Stage 2 average. Table 1 reports 57.57 Overall AVG, compared with 56.98 for the corresponding OLMo-3-7B model, with 83.02 GSM8K and 68.48 MMLU.
• Three matched-budget data recipes. Appendix C compares V1, V2, and V3 under a common inference and sampling setup. V1 leads on the reported code, mathematics, academic-STEM, and logical-reasoning tasks; V2/V3 lead on HellaSwag.
• A Target for concept-conditioned drafting. Stage 2 V1 is the paired Target for the separate NCPFlash draft model.

The report also finds that Stage 2 reaches the OLMo-3-7B final training loss with 66.2% of the training tokens, corresponding to 1.51x convergence in token budget. This is a training-loss comparison, not an inference throughput measurement.

Stage 2 Releases

These labels identify alternative Stage 2 data recipes. Higher version numbers do not imply a stronger model: although aggregate training loss decreases from V1 to V3, several downstream capabilities decline. The quick-start example uses V1, which is the strongest of these recipes across the free-form capabilities evaluated in Appendix C.

Evaluation

Main Stage 2 Comparison

The following values are the report's main Stage 2 comparison with OLMo-3-7B and should not be assigned to all three released variants. Scores are percentages; deltas are absolute percentage points.

Likelihood is reported separately in bits per UTF-8 byte (BPB), where lower is better.

The main Stage 2 model improves the overall average while showing a mixed profile across individual tasks, including lower HumanEval and ARC-Challenge scores. The report discusses a possible mismatch between the continued-training mixture and downstream domains; lower aggregate loss alone does not establish better performance on every task.

V1 / V2 / V3 Recipe Comparison

All three Stage 2 variants share the same inference and sampling configuration within this comparison.

For recipe selection, the report treats V2 and V3 as effectively tied on HellaSwag because they differ by only 0.05 percentage points. Its held-out proxy analysis supports the V1 > V2 > V3 ordering on four free-form capability measurements. This evidence compares complete recipes and does not isolate the contribution of an individual data source.

Protocol and Aggregation

The main comparison follows the OLMo evaluation protocol in OLMo-Core. Appendix E and Table 13 specify the per-task settings, including:

• GSM8K: 8-shot, one completion per problem, pass@1.
• MATH-500: 4-shot, 32 completions per problem, pass@1.
• HumanEval and MBPP: 3-shot, 32 completions per problem, execution pass@1.

The sample count is the number of generated completions per problem, not the inference batch size. Overall AVG is the unweighted mean of 26 constituent benchmark scores, excluding the aggregate MMLU row and domain averages. The ten-benchmark BPB AVG is calculated separately and is excluded from Overall AVG. See the evaluation configuration for the corresponding release assets