SenseNova-Vision-MoT

🌟 Overview

SenseNova-Vision is a unified multimodal model for computer vision. It reformulates heterogeneous visual perception tasks as text generation, image generation, or mixed text-image generation, instead of relying on task-specific heads, decoders, or loss functions for each individual task. The model supports structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry within a shared instruction-following interface.

🚀 Model Description

SenseNova-Vision rethinks computer vision as unified multimodal generation. Traditional computer-vision systems usually attach task-specific prediction heads for detection, segmentation, depth, surface normal, or 3D geometry. SenseNova-Vision instead expresses these heterogeneous tasks through the native input-output spaces of a unified multimodal model.

Natural-language instructions and optional visual prompts specify the target task, regions, views, output schema, and decoding convention. The model then generates different target formats depending on the task:

This formulation allows a single model to cover structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry while keeping outputs decodable for standard benchmarks.

🌐 Key Features

• Unified vision-task formulation: Heterogeneous computer-vision tasks are cast into the native text, image, and mixed generation spaces of a unified multimodal model.
• No task-specific heads: The model does not rely on separate detection, segmentation, depth, normal, or geometry heads.
• Decodable outputs: Generated text and images can be converted back into benchmark-compatible boxes, points, OCR strings, masks, depth maps, normal maps, point maps, and camera records.
• Broad task coverage: The same model handles structured visual understanding, segmentation, dense geometry, and multi-view visual geometry.
• Instruction-defined task variants: Natural-language instructions enable flexible task definitions beyond fixed benchmark schemas.

🛠️ How to Use

Please use the official inference code from the SenseNova-Vision GitHub repository:

git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
cd SenseNova-Vision

Environment Setup

Create the environment from the repository root:

bash setup.sh sensenova-vision
conda activate sensenova-vision

Download the Model

You can download the model weights from Hugging Face with huggingface_hub:

from huggingface_hub import snapshot_download

model_path = snapshot_download("sensenova/SenseNova-Vision-7B-MoT")
print(model_path)

The printed model_path points to the local checkpoint directory and can be used as the model path for inference.

Run the Curated Example

We provide a curated example to quickly verify the environment and model setup:

bash scripts/run_sensenova_vision.sh example

Run One Inference Request

You can also run a single inference request with the official wrapper. For example, the following command performs binary segmentation for the target category "person":

bash scripts/run_sensenova_vision.sh inference \
  binary_seg \
  "person" \
  examples/images/2.jpg

Launch the Web Demo

You can launch the Gradio web demo using the official wrapper provided in the repository. The wrapper will print the local URL before starting Gradio. Open the printed URL in your browser to interact with the model.

For more details, supported tasks, and additional examples, please refer to the official GitHub repository:

https://github.com/OpenSenseNova/SenseNova-Vision

🏗️ Key Contributions

• 🔗 We introduce a unified multimodal generation formulation that casts heterogeneous computer vision tasks into the native input-output spaces of UMMs.
• 🧩 We construct the SenseNova-Vision Corpus, a large-scale computer-vision instruction-response corpus with decodable text, image, and mixed text-image targets.
• ✨ We train SenseNova-Vision and show strong results across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry, while supporting language-defined task variants beyond fixed benchmark schemas.

🏆 Benchmark Results

SenseNova-Vision is evaluated across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. All tasks are formulated with natural-language instructions: textual outputs are parsed into benchmark-specific structures such as boxes, points, recognized text, keypoints, and camera parameters, while image outputs are decoded into masks, depth maps, normal maps, or 3D point maps.

Structured Visual Understanding

Structured visual understanding evaluates tasks whose outputs can be represented as structured textual predictions, including box- and point-based localization, referring detection, OCR localization, GUI grounding, and keypoint localization.

Dense Geometric Prediction

Dense geometric prediction evaluates pixel-aligned geometric outputs, including monocular depth estimation and surface normal estimation.

Segmentation

Segmentation evaluates mask prediction under semantic, referring, reasoning, grounded, and interactive guidance.

Multi-View Visual Geometry

Multi-view visual geometry evaluates geometric prediction from multiple input images, including multi-view point map reconstructi