omni-embed-nemotron

Description

NV-QwenOmni-Embed-3B-v1 is a versatile multimodal embedding model capable of encoding content across multiple modalities, including text, image, audio, and video, either individually or in combination, and supports retrieval using queries that can also be multimodal. It is designed to serve as a foundational component in multi-modal Retrieval-Augmented Generation (RAG) systems.

The foundational Qwen Omni model (Qwen/Qwen2.5-Omni-3B) is based on the Thinker-Talker architecture. We only leverage the Thinker component to encode and understand diverse modalities. In this implementation, we do not include the Talker component, as the model focuses on multimodal understanding rather than response generation.

This model is for research and development only.

For more technical details, please refer to our technical report: Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video

License/Terms of Use

Governing Terms for nvidia/omni-embed-nemotron-3b model: NVIDIA OneWay Noncommercial License.

ADDITIONAL INFORMATION: Qwen RESEARCH LICENSE AGREEMENT

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Team

• Mengyao Xu
• Gabriel Moreira
• Radek Osmulski
• Ronay Ak
• Yauhen Babakhin
• Bo Liu
• Even Oldridge
• Benedikt Schifferer

Citation

@article{xu2025omni,
  title={Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video},
  author={Xu, Mengyao and Zhou, Wenfei and Babakhin, Yauhen and Moreira, Gabriel and Ak, Ronay and Osmulski, Radek and Liu, Bo and Oldridge, Even and Schifferer, Benedikt},
  journal={arXiv preprint arXiv:2510.03458},
  year={2025}
}
@misc{moreira2025nvretrieverimprovingtextembedding,
      title={NV-Retriever: Improving text embedding models with effective hard-negative mining},
      author={Gabriel de Souza P. Moreira and Radek Osmulski and Mengyao Xu and Ronay Ak and Benedikt Schifferer and Even Oldridge},
      year={2025},
      eprint={2407.15831},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2407.15831},
}

Deployment Geography

Global

Use Case

NV-Omni-Embed is intended for researchers and developers building retrieval-based applications that require understanding and retrieve information across multiple modalities. It is particularly useful in multimodal RAG systems, where queries and documents may include combinations of text, images, audio, and videos. Potential applications include multimedia search engines, cross-modal retrieval systems, and conversational AI with rich input understanding.

Release Date

Huggingface 10/1/2025 via [https://huggingface.co/nvidia/omni-embed-nemotron-3b]

Model Architecture

• Architecture Type: Transformer
• Network Architecture: Qwen/Qwen2.5-Omni-3B

NV-QwenOmni-Embed-3B-v1 is a transformer-based multimodal embedding model built on top of the Thinker component from Qwen/Qwen2.5-Omni-3B. Unlike the original Thinker-Talker architecture, this model does not include the Talker module, as it is designed specifically for multimodal understanding and retrieval rather than response generation. Number of model parameters is 4.7B.

The model incorporates a vision encoder, an audio encoder, and a large language model (LLM) from the Qwen architecture to process diverse modalities. Unlike the Omni model, which interleaves audio and video tokens with TMRoPE, our retrieval encoder keeps the two streams separate. Audio and video are encoded independently, preserving their full temporal structure without interleaving. Our experiments show this design improves retrieval performance.

NV-QwenOmni-Embed-3B-v1 is trained using a bi-encoder architecture where queries and candidate inputs are embedded independently. A contrastive learning objective is employed to align relevant query-content pairs while pushing apart unrelated ones in the shared embedding space.

Input

Other Properties: The model's maximum context length is 32768 tokens.

Output

• Output Type: Floats
• Output Format: List of float arrays
• Output Parameters: A tensor of floats equivalent to [batchsize x 2048]
• Other Properties Related to Output: Model outputs embedding vectors of dimension 2048 for each input.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Usage

Using Sentence Transformers

Install Sentence Transformers:

pip install sentence_transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "nvidia/omni-embed-nemotron-3b",
    trust_remote_