NV-Reason-CT

NV-Reason-CT is a 3D vision-language model (VLM) for CT image analysis. It combines a native 3D vision encoder (3D ViT) with a language model and is designed for radiology report generation, general question answering, and multi-step reasoning across chest and abdominal CT volumes.

Computed tomography encodes clinically important anatomy across hundreds of slices, yet most vision–language systems either operate on 2D images or compress volumetric features before language decoding. The 3D vision encoder converts a 384×384×384-mm input volume into a 24×24×24 grid of 13,824 visual tokens. All tokens and their corresponding 3D positions are passed to the language model without spatial downsampling, while 3D MRoPE preserves their spatial relationships within the LLM. Input volumes are automatically cropped to the chest or abdomen and resampled to 2-mm isotropic resolution before processing.

The model was trained end-to-end using Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on a curated dataset of approximately 550,000 structured QA examples from 70,111 unique CT volume inputs. The training corpus integrates standardized reports, abnormality-focused QA, multi-turn interactions, and radiologist-authored reasoning collected through recorded and transcribed expert CT interpretations. These expert annotations also guide the generation of report-grounded synthetic reasoning data for SFT, while GRPO uses verifiable rewards over chest and abdominal abnormality sets.

Installation

Python 3.11+ and a CUDA-capable PyTorch installation are recommended.

python -m pip install -r https://huggingface.co/nvidia/NV-Reason-CT/resolve/main/requirements.txt

Quick Start

The example below loads the model once and defines a reusable function for deterministic inference on .nii.gz volumes. Use anatomy_region="chest" for a chest crop or anatomy_region="abdomen" for an abdominal crop.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "nvidia/NV-Reason-CT"
ct_path = "path/to/volume.nii.gz"

model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
).eval().to("cuda")

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

def generate_response(
    ct_path,
    prompt_text,
    anatomy_region="chest",
    enable_thinking=True,
    max_new_tokens=2048,
):
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": prompt_text},
            ],
        }
    ]
    prompt = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=enable_thinking,
    )
    inputs = processor(
        text=prompt,
        images3d=[ct_path],
        anatomy_region=anatomy_region,
        return_tensors="pt",
    ).to(model.device)

    with torch.inference_mode():
        generated_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            use_cache=True,
        )

    new_tokens = generated_ids[:, inputs.input_ids.shape[1]:]
    return processor.batch_decode(
        new_tokens,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]

# Structured report (Chest region)
print(generate_response(ct_path, "Write a structured chest CT report.", anatomy_region="chest"))

The same loaded model can be reused with other prompts and anatomy regions.

# Structured report (Abdominal region)
print(generate_response(ct_path, "Write a structured abdominal CT report.", anatomy_region="abdomen"))

# Reasoning (Chest region)
print(generate_response(ct_path, "Provide a full reasoning analysis of this chest CT.", anatomy_region="chest"))

# Reasoning (Abdominal region)
print(generate_response(ct_path, "Provide a full reasoning analysis of this abdominal CT.", anatomy_region="abdomen"))

# Binary question with reasoning
print(generate_response(ct_path, "Is a pleural effusion present in this CT?", anatomy_region="chest"))

# Concise Yes/No response without thinking
print(generate_response(
    ct_path,
    "Is a pleural effusion present in this CT? Answer only Yes or No.",
    anatomy_region="chest",
    enable_thinking=False,
))

Anatomy region cropping

AutoProcessor includes anatomy-aware cropping around the chest or abdomen using lung Hounsfield units and a 3D morphology heuristic. Only "chest" and "abdomen" are supported. This works reasonably well for common CT geometries, such as whole-body CT, chest plus upper abdomen, or lower chest plus abdomen and pelvis. You can inspect the exact crop passed to the VLM:

import nibabel as nib
import numpy as np
from transformers import AutoProcessor

model_id = "nvidia/NV-Reason-CT"
ct_path = "path/to/volume.nii.gz"
anatomy_region = "chest"  # "chest" or "abdomen"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

# Run the same 2 mm resampling and anatomy-aware 192 x 192 x 192 crop used
# for inference. normalize_mode=0 preserves CT Hounsfield units.
cropped_image = processor.image_processor_3d.load_image(
    ct_path,
    normalize_mode=0,
    anatomy_region=anatomy_region,
)

# load_image() returns a channel-first tensor in (C, Z, Y, X) order for the
# model. Convert the spatial axes back to NIfTI (X, Y, Z) order before saving.
cropped_volume = cropped_image[0].permute(2, 1, 0).detach().cpu().numpy()
affine = np.diag([-2.0, -2.0, 2.0, 1.0])
cropped_nifti = nib.Nifti1Image(
    cropped_volume,
    affine,
)

# Inspect the cropped image; t