NLD-Uni

Description

NLD-Uni is a unified multimodal model for image understanding and image generation. It combines a diffusion language model with specialized visual components and a mixture-of-transformers design. Understanding and generation use distinct transformer parameter paths within a common multimodal framework.

For image understanding, NLD-Uni encodes images into continuous visual features and generates text responses. For image generation, it models discrete visual tokens through iterative diffusion and decodes those tokens into RGB images. The architecture supports autoregressive and blockwise diffusion decoding for text, as well as image-conditioned generation and editing when enabled by the checkpoint and inference implementation.

This card describes the NLD-Uni architecture generally.

Environment

Install the dependencies in requirements.txt with Python 3.13 and CUDA 12.6. Install PyTorch before building FlashAttention:

pip install -r requirements.txt
pip install --no-build-isolation flash-attn==2.8.0.post2

Validated on an NVIDIA H100 80 GB. The eight BF16 weight shards contain all 1,557 active model tensors. weights_manifest.json records their hashes.

Python inference

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    '/path/to/hf_checkpoint',  # or 'organization/model-name'
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to('cuda').eval()

image = model.text_to_image('A red ceramic mug on a wooden table.', seed=42)
image.save('mug.png')

edited = model.image_edit(image, 'Change the mug to blue.', seed=42)
edited.save('blue_mug.png')

answer = model.generate('What color is the mug?', image=edited)
print(answer)

Image inputs accept Pillow images or local file paths. Image methods return Pillow images; generate returns text by default. Token IDs can be supplied through input_ids for text prompts, and return_token_ids=True returns the generated token tensor. The inference.py command exposes each task.

Minimal command-line inference

python inference.py /path/to/hf_checkpoint --task t2i --prompt 'A red mug.' --output mug.png
python inference.py /path/to/hf_checkpoint --task edit --image mug.png --prompt 'Make the mug blue.' --output blue.png
python inference.py /path/to/hf_checkpoint --task understand --image blue.png --prompt 'What color is the mug?'

Replace the model argument with the published Hugging Face model ID for remote loading. The standalone script only uses Transformers and this repository's model code.

License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Source License Agreement.

Deployment Geography

Global, subject to the governing release terms.

Use Case

NLD-Uni is intended for research and evaluation of unified multimodal systems, including:

• Visual question answering and image-grounded reasoning.
• Document, chart, and scene-text understanding.
• Visual grounding and interface localization using task-specific prompts and output formats.
• Text-to-image generation.
• Image-conditioned generation and editing for checkpoints trained for these tasks.

Release Date

10/27/2026

Model Architecture

Architecture Type: Transformer

Network Architecture: Diffusion language model with understanding and generation transformer paths, a continuous visual encoder for understanding, and a discrete image tokenizer for generation.

Language-model foundation: Nemotron diffusion language model.

Input

Input Type(s): Text and images.

Input Format(s): Text strings and image objects or tensors accepted by the supplied processor.

Input Parameters: One-dimensional text-token sequences and two-dimensional image content, represented internally as visual features or discrete image tokens according to the task.

Other Properties Related to Input: Context length, image resolution, visual-token budget, and preprocessing are checkpoint- and task-dependent. The released inference configuration defines the supported limits; a universal 900-token prompt limit is not specified for the architecture.

Output

Output Type(s):

• Text
• Image

Output Format(s):

• Text: .json (UTF-8 JSON records containing generated text).
• Image: .png (RGB images).

These serialized formats are implemented by validate.py. The inference API returns text strings or token IDs for understanding and decoded image objects or tensors for generation. Grounding outputs follow the requested coordinate or location-token format.

Output Parameters: One-dimensional token sequences or images with height, width, and three color channels. Tensor layout and pixel range are defined by the inference API.

Other Properties Related to Output: Text output length and generated image resolution depend on the checkpoint and inference settings. Configurations for 1,024 × 1,024 image generation are implemented; the release should list its validated resolutions.

Software Integration

NLD-Uni’s native implementation uses PyTorch with CUDA-accelerated execution. The codebase also includes a dedicated SGLang integration for the image-understanding and text-generation path. This does not establish SGLang support for the image-generation path.

Runtime Engine(s):

• PyTorch for native multimodal inference.
• SGLang with the NLD-Uni/LaVida RADIO-H adapter for supported understanding configurations.

Reference Platform: Linux with NVIDIA CUDA GPUs.

Reference Evaluation Hardware: NVIDIA H100 (Hopper).

Model Version(s)

• NLD-Uni: Unified image-understanding and image-generation architecture based on LaVida-O.

Training stage, checkpoint revision, tokenizer revision, and inference configuration are part of the identity of an evaluated model. Results from earlier LaVida-O or image-only diffusion mo