NVIDIA-Nemotron-Labs-Dory1.0

Model Summary

NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4 is a looped (latent-recurrent) large language model (LLM) trained by NVIDIA. The model employs a normalized-GPT architecture with interleaved hybrid sliding window attention (SWA) and full attention layers. By repeatedly applying a shared block of layers, Dory1.0-4B can spend more compute per token allowing it to scale intelligence. This model is ready for commercial or non-commercial use.

Highlights

• Elastic Inference Compute: A single checkpoint serves different inference compute budgets. By changing how many times the shared block of layers is applied, you can trade inference compute for accuracy to suit each task, with no retraining and no model swap.
• Token Efficiency: With looping, the model needs fewer generated tokens to solve a task. Extra computation per token goes into thinking in latent space rather than into longer outputs.
• Memory-Efficient Looping: Standard looping shares parameters across loops but still stores a separate KV cache for each one, so memory grows with every added loop. Dory supports last-loop KV caching, which keeps a single cache per recurrent attention layer. When enabled, Dory carries no KV-cache overhead relative to an equivalent non-looped model, regardless of the inference loop count.
• Native NVFP4 Pre-training: Built on a normalized-GPT (nGPT) backbone, Dory is pre-trained natively in NVFP4, NVIDIA's 4-bit floating-point format, before continuing in BF16.
• Multiple Formats: The model is available in BF16, MXFP8, NVFP4 and Q5_K_M formats.

License/Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.

Benchmarks

(1) AA-LCR: We use the corrected answer keys (from AA-LCR v1.1) but keep the older judge (Qwen3-235B-A22B) instead of GPT-5.6 Luna. (2) GDPval-AA v2: Elo ratings of selected reference models (out of 12): DeepSeek-V4-Pro (1223), Qwen3.6-35B-A3B (992), Qwen3.5-122B-A10B (925), gpt-oss-120b (745). (3) Repeats: Number of repeated runs per benchmark: AA-Briefcase-Lite (1), GDPval-AA v2 (1), AutomationBench-AA (1), Terminal-Bench 4.0 (1), SciCode (8), Humanity's Last Exam (1), GDP.pdf text-only (5), AA-Omniscience (10), AA-LCR (16).

Quick Start Guide

vLLM

Note: Dory support in vLLM is under review in vllm-project/vllm#60149. Until it is included in a vLLM release, install vLLM from that PR:

VLLM_USE_PRECOMPILED=1 pip install "git+https://github.com/vllm-project/vllm.git@refs/pull/60149/head"

VLLM_USE_PRECOMPILED=1 reuses compatible prebuilt vLLM kernels, avoiding local CUDA compilation. This section will be updated once a vLLM release supports Dory.

Launch the Server

vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4 \
    --hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
    --max-model-len 262144 \
    --max-num-seqs 128 \
    --enable-prefix-caching \
    --async-scheduling \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Validated context: 256K tokens. Lower --max-model-len if you are memory-constrained or want more KV-cache headroom at higher concurrency.

Choosing the Loop Count

We recommend two loops as the default starting point for a good balance between accuracy and compute. Most of the accuracy gain from looping comes from the second loop, with smaller gains beyond that (see the loop-count comparison).

Use one loop when minimizing compute per token is the priority. Consider four loops when additional accuracy matters more, particularly for agentic tasks such as Terminal-Bench and SWE-bench (see benchmark results). Because higher loop counts can also reduce the number of generated tokens, evaluate the end-to-end latency and cost on your workload when choosing a setting.

To use a different loop count, update n_recurrent_loops in --hf-overrides in the command above.

Reducing the KV Cache Memory Overhead

The command above uses per-loop caching, which stores a separate KV cache for each recurrent loop, so KV cache memory grows with the loop count. Last-loop KV caching instead keeps a single cache per recurrent attention layer where each loop overwrites the previous loop's entries, so the cache holds only the final loop's keys and values. This keeps the KV cache footprint at the one-loop size for any loop count, leaving more capacity for longer contexts or concurrent requests.

To enable it, set recurrent_kv_cache_mode to "last_loop". In our experiments, the accuracy with last-loop caching stays close to the per-loop mode with nearly unchanged output lengths.

llama.cpp

Note: Dory support in llama.cpp is available in ggml-org/llama.cpp#30306. Until it is included in a llama.cpp release, build from the Dory-enabled source below. This example pins revision 98a74c97ea09ac72f6ceda1559808fc0a2f447ce, used by the local macOS reference setup, rather than following the moving PR head.

Build on Apple Silicon

These commands target native ARM64 macOS on Apple Silicon, not Intel or Rosetta. Install Homebrew and Apple's Command Line Tools (`