Cosmos-H-Surgical-Simulator

Description

Cosmos-H-Surgical-Simulator is a kinematic action-conditioned surgical world foundation model, built on the public NVIDIA Cosmos-Predict2.5-2B for physical AI and fine-tuned on the Open-H multi-embodiment surgical benchmark. Unlike the text-conditioned base model, it is driven directly by robot kinematics: given a surgical context frame and a sequence of 44-dimensional action vectors encoding end-effector poses and gripper commands (unified across 9 embodiments), it generates future video of the resulting surgical scene.

The model is intended for evaluating surgical robotics policies in simulation and for synthetic data generation prior to deployment on a physical system. It covers CMR Surgical Versius clinical procedures (cholecystectomy, prostatectomy, inguinal hernia, hysterectomy) as well as dVRK, MITIC, and other surgical platforms across tasks such as suturing, tissue manipulation, and peg transfer.

This model is for commercial/non-commercial use.

Updates

• April 2026 — Released updated checkpoint after fixing an action-embedder MLP initialization bug. Aggregate quality improves on all three metrics: FDS (L1) 0.223 → 0.184 (−17%), GATC 0.417 → 0.472 (+13%), TCD 83.68 → 67.03 (−20%).

License/Terms of Use

Use of this model is governed by the NVIDIA Open Model License Agreement.

Deployment Geography

Global

Use Case

Primarily intended for surgical robotics researchers, healthcare AI developers, academic institutions, and surgical robotics companies, exploring surgical robotics policy evaluation and synthetic data generation.

Release Date

• GitHub: 3/13/2026 via https://github.com/NVIDIA-Medtech/Cosmos-H-Surgical-Simulator
• Huggingface: 3/15/2026 via https://huggingface.co/NVIDIA

Reference(s)

Ali, A., Bai, J., Bala, M., Balaji, Y., Blakeman, A., Cai, T., Cao, J., Cao, T., Cha, E., Chao, Y.-W., Chattopadhyay, P., Chen, M., Chen, Y., Cheng, S., Cui, Y., Diamond, J., Ding, Y., Fan, J., Fan, L., Feng, L., Ferroni, F., Fidler, S., Fu, X., Gao, R., Ge, Y., Gu, J., … Zhu, Y. (2025).World Simulation with Video Foundation Models for Physical AI (arXiv:2511.00062) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2511.00062

Link to Cosmos’ nvidia/Cosmos-Predict2.5-2B-Video2World Model Card

Model Architecture

Architecture Type: Diffusion Transformer Network Architecture: Latent video diffusion transformer (DiT-style denoiser) with cross-attention conditioning.

This model was developed based on Cosmos-Predict2.5-2B-Video2World.

The Cosmos-H-Surgical-Simulator model extends Cosmos-Predict2.5-2B-Video2World, a diffusion transformer for video generation in latent space. It incorporates an MLP to condition the model on kinematic actions. The model accepts a 44-dimensional action vector (22 dimensions per arm) alongside the current video frame, and predicts the subsequent 12 frames. Through autoregressive rollout, it can generate videos of complete surgical trajectories from either learned policies or manually designed action sequences.

Input

• Input Type(s): Image (camera frame), a sequence of twelve 44-dimensional numerical vectors
• Input Format(s): Red, Green, Blue (RGB) frame, numeric vector
• Input Parameters: Image: Two-Dimensional (2D) image frame , Vector: Sequence of 12 forty-four-dimensional (44D) vectors Other Properties Related to Input:
• Recommended resolution: The model operates at 512x288 resolution. Input frames are automatically resized; a 16:9 aspect ratio is recommended.
• Pre-processing: The Versius kinematic action is a hybrid relative action as defined here.

Output

• Output Type(s): A sequence of 12 video frames
• Output Format: mp4
• Output Parameters: Three-Dimensional (3D)
• Other Properties Related to Output:
  • By default, the model generates twelve video frames as output, representing the next world states incorporating the input kinematic actions into the current video frame. Through an autoregressive loop, a complete video sequence, representing a complete surgical trajectory, can be generated.
  • No additional post-processing is strictly required.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s):

Cosmos-Predict2.5

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper

Note: Only BF16 precision is tested. Other precisions like FP16 or FP32 are not officially supported.

Preferred/Supported Operating System(s): Linux (We have not tested on other operating systems.)

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

v1.0 (Finetuned on the Open-H Embodiment dataset, which includes clinical procedures data such as cholecystectomy, prostatectomy, inguinal hernia, and hysterectomy)

Developers may integrate the model into an AI evaluation system by providing video frames as input along with corresponding kinematic actions to evaluate a surgical policy model, such as one for the CMR Surgical Versius robotic system.

Training, Testing, and Evaluation Datasets

Da