Nemotron-3-Labs-Ultra-Math-SFT

Referred to as Nemotron-3-Ultra-SFT in the technical report.

Description

Nemotron-3-Labs-Ultra-Math-SFT is a decoder-only transformer language model specialized for mathematical reasoning, trained to solve difficult mathematical problems and identify mistakes in proofs, and deployed as part of an ensemble system that achieved a gold-medal level score at the International Mathematical Olympiad 2026.

Full details can be found at our technical report An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics.

Nemotron-3-Labs-Ultra-Math-SFT was developed by NVIDIA as a part of Nemotron.

This model is ready for commercial and non-commercial use.

License/Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1 (OpenMDW-1.1).

Deployment Geography

Global

Use Case

Researchers and developers focused on AI-driven mathematical reasoning and proof verification, aiming to advance open models for solving complex math problems and improving reasoning capabilities.

Release Date

HuggingFace: September 3, 2026 via https://huggingface.co/collections/nvidia/nemotron-labs-imo-2026

Reference(s)

• An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics (technical report)
• nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
• nvidia/Nemotron-Math-Proofs-v3-SFT
• nvidia/Nemotron-IMO-Bench
• Inference pipeline and submitted proofs: https://github.com/NVIDIA-NeMo/Skills/tree/main/recipes/nemotron-imo-tts
• RL training recipe: https://github.com/NVIDIA-NeMo/RL/blob/imo-26-ultra-v3/docs/guides/nemotron-3-ultra-imo.md

Model Architecture

Architecture Type: Mamba2-Transformer Hybrid Latent Mixture of Experts (LatentMoE) with Multi-Token Prediction (MTP) Network Architecture: Nemotron Hybrid LatentMoE This model was developed based on nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16. Number of model parameters: 550B Total / 55B Active

Input

Input Type(s): Text Input Format(s): String Input Parameters: One-Dimensional (1D) Other Properties Related to Input: Maximum context length up to 1M tokens

Output

Output Type(s): Text Output Format: String Output Parameters: One-Dimensional (1D) Other Properties Related to Output: Maximum context length up to 1M tokens

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s): vLLM Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
• NVIDIA Hopper

Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

Nemotron-3-Labs-Ultra-Math-SFT v1

Deployment

The Ultra BF16 checkpoint is a frontier-scale model. The minimum recommended hardware is:

• Single-node: 8× B200 (≈1.5 TB aggregate HBM — fits BF16 weights plus KV cache with headroom)
• Multi-node: ≥8 GPUs across H100 / H200 / GB200 / GB300, orchestrated with Ray v2

For more detailed information, please see the nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 model card and this cookbook.

Recommended container: vllm/vllm-openai:v0.22.0.

export MODEL_CKPT=PATH/TO/MODEL/CHECKPOINT

8× B200 single-node deployment:

docker run -d --name nemotron-ultra-vllm \
  --gpus all \
  --ipc=host \
  --network=host \
  --shm-size=16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v $MODEL_CKPT:/model:ro \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e SAFETENSORS_FAST_GPU=1 \
  -e NVIDIA_TF32_OVERRIDE=1 \
  -e VLLM_LOGGING_LEVEL=INFO \
  vllm/vllm-openai:v0.22.0 \
  /model \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name nvidia/Nemotron-3-Labs-Ultra-Math-SFT \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mamba-ssm-cache-dtype float16 \
  --mamba-backend flashinfer \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}' \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 96}'

Context length defaults to 256k above. To use up to 1M, set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 and --max-model-len 1048576.

Training, Testing, and Evaluation Datasets

Training Dataset

Data Modality: Text

Text Training Data Size: 39,047,191,715 recorded