Cosmos3-H-Dreams

Description

Cosmos3-H-Dreams is a real-time, action-conditioned surgical world model for the CMR Surgical Versius robotic system. It is fine-tuned on CMR Surgical Versius clinical cholecystectomy data and distilled into a causal, few-step student for interactive, streaming video generation. Given a single endoscopic camera frame, a short history of recent robot commands and a stream of future kinematic and control commands for both arms, the model generates the resulting endoscopic video autoregressively.

The model is intended for surgical robotics policy evaluation in simulation, interactive (teleoperated) world-model exploration and synthetic data generation for robotic cholecystectomy, before transitioning to a physical system.

Cosmos3-H-Dreams is the successor of Cosmos-H-Dreams (arXiv:2608.24199), which was based on Cosmos-Predict2.5. Cosmos3-H-Dreams is based on Cosmos3-H-Embodied-Nano, which is in turn based on the NVIDIA Cosmos3-Nano world foundation model for physical AI.

This model is ready for commercial or non-commercial use.

License/Terms of Use

GOVERNING DOWNLOAD TERMS: Use of the model is governed by the NVIDIA Open Model License Agreement.

Additional Information: Use of the Cosmos3-H-Embodied-Nano base model is governed by the NVIDIA Open Model License Agreement.

Deployment Geography

Global

Use Case

Primarily intended for surgical robotics researchers, healthcare AI developers, academic institutions and surgical robotics companies exploring real-time surgical world models, surgical robotics policy evaluation and synthetic data generation for the CMR Surgical Versius system.

Release Date

• GitHub: 10/20/2026 via https://github.com/isaac-for-healthcare/Cosmos3-H-Dreams
• Huggingface: 10/20/2026 via https://huggingface.co/nvidia/Cosmos3-H-Dreams

Reference(s)

Cosmos-H-Dreams (predecessor model): arXiv:2608.24199

Direct base model: nvidia/Cosmos3-H-Embodied-Nano

Base model of Cosmos3-H-Embodied-Nano: nvidia/Cosmos3-Nano

Self-Forcing (causal video diffusion distillation): Huang, X., Li, Z., He, G., Zhou, M., Shechtman, E. (2025). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (arXiv:2506.08009) [Preprint]. arXiv. https://arxiv.org/abs/2506.08009

Distribution Matching Distillation: Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T. (2024). Improved Distribution Matching Distillation for Fast Image Synthesis (arXiv:2405.14867) [Preprint]. arXiv. https://arxiv.org/abs/2405.14867

Model Architecture

Architecture Type: Mixture-of-Transformers (MoT) Network Architecture: Cosmos3-Nano unified Mixture-of-Transformers (a Qwen3-VL-8B-Instruct reasoning tower plus a latent-diffusion generation expert with shared multimodal attention), inherited through Cosmos3-H-Embodied-Nano, operating on Wan2.2 VAE latents (4x temporal, 16x16 spatial compression), with action embedding/projection layers for kinematic conditioning; distilled from a bidirectional teacher into a block-causal, 4-step autoregressive student with a KV cache.

This model was developed based on Cosmos3-H-Embodied-Nano (which is based on Cosmos3-Nano).

Number of model parameters: ~1.5 × 10^10 (15.17B)

Cosmos3-H-Dreams was produced in three phases:

1. Initialization from Cosmos3-H-Embodied-Nano: Cosmos3-H-Embodied-Nano is a Cosmos3-Nano action-conditioned checkpoint post-trained on the Open-H-Embodiment dataset for multiple surgical robot embodiments, with up to 16 past action transitions as clean action history. It provides the initialization.
2. CMR cholecystectomy teacher fine-tuning: The action interface for the CMR Versius embodiment (embodiment domain 25 of 32) was extended from 20 to 28 dimensions per time step by adding electrosurgical energy and thumbstick controls for both arms. Pretrained pose/jaw weights were preserved and the new control inputs were zero-initialized. A bidirectional teacher was fine-tuned on Versius cholecystectomy data first with a 24-frame horizon (2,000 iterations), then with a 72-frame horizon (4,400 iterations).
3. Real-time distillation: The bidirectional 72-frame teacher (35 sampling steps) was distilled into a block-causal student using causal teacher forcing (1,000 iterations), followed by Self-Forcing with distribution-matching distillation (300 iterations) against the fixed bidirectional teacher. The resulting student generates video autoregressively with a KV cache and 4 denoising steps per latent frame.

At inference, the conditioning frame and the optional action history (up to 16 transitions, 1.6 s) are prefilled into the KV cache once. Each subsequent latent frame consumes the next 4 future action vectors (0.4 s at 10 Hz) and produces 4 new video frames, enabling streaming rollout.

Input

• Input Type(s): Image (camera frame), numerical action vectors (future time steps)
• Input Format(s): Red, Green, Blue (RGB) frame, numeric vectors
• Input Parameters: Image: Two-Dimensional (2D) image frame; Vector: One-Dimensional (1D) sequence of 28-dimensional (28D) action vectors (up to 16 history + 72 future per rollout window, at 10 Hz)
• Other Properties Related to Input:
• Recommended resolution: The model operates at 832x480. A 16:9-like aspect ratio is recommended; input frames are resized to the model resolution.
• Action vector: 14 dimensions per arm (left arm 0–13, right arm 14–27): anchor-relative translation (3), anchor-relative 6D rotation (6), jaw (1), electrosurgical energy active (1), thumbstick button (1), thumbstick axes (2