Trident-Qwen3VL

Official model for Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

Trident-Qwen3VL-2B is a multimodal embedding model for mixed-modality retrieval. It maps text, images, and fused text-image inputs into one shared space, so a text query can rank text-only, image-only, and fused documents in a single list. It is fine-tuned from Qwen3-VL-2B-Instruct on Trident-Train, as described in the paper Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers. The LoRA adapter is already merged, so no base model or PEFT is needed.

Quickstart

1. Set up the code and download the weights

git clone https://github.com/OpenBMB/Trident.git
cd Trident
conda create -n trident python=3.12 -y && conda activate trident
pip install -r requirements.txt     # tested with transformers 4.57.1 (Qwen3-VL needs >= 4.57)

hf download OpenBMB/Trident-Qwen3VL-2B --local-dir checkpoints/Trident-Qwen3VL-2B

2. Encode queries and documents (run from the root of the Trident repository)

import sys
sys.path.insert(0, "src")

from embedding_backends.common import (
    ensure_tokenizers_thread_safety,
    ensure_transformers_special_tokens_compat,
)
from embedding_backends.trident_qwen3vl import TridentQwen3VLBackend

ensure_transformers_special_tokens_compat()
ensure_tokenizers_thread_safety()

CKPT = "checkpoints/Trident-Qwen3VL-2B"

backend = TridentQwen3VLBackend(
    {"model_type": "trident_qwen3vl", "checkpoint": CKPT, "dtype": "bfloat16"},
    device="cuda:0",
)
backend.load()

def encode(items, role):
    """items: list of {"text": str | None, "images": [paths]}; role: "query" or "doc"."""
    backend.args_dict["role"] = role
    instruction = backend.default_instruction(role)
    inputs = [{**item, "instruction": instruction} for item in items]
    embeddings, _ = backend.process(inputs, role=role)
    return embeddings  # [N, dim] numpy array, L2-normalised

queries = [
    {"text": "How many stores did Saint Laurent operate in Western Europe in 2020?", "images": []},
]

documents = [
    {"text": "A bar chart showing the number of Saint Laurent stores by region and year.", "images": []},  # text-only
    {"text": None, "images": ["figs/example_chart.png"]},                                                # image-only
    {"text": "Number of stores by region.", "images": ["figs/example_chart.png"]},                       # text + image
]

q = encode(queries, role="query")      # [1, dim]
d = encode(documents, role="doc")      # [3, dim]

scores = q @ d.T                       # cosine similarity, comparable across all document types
print(scores)
print(scores[0].argsort()[::-1])       # document ranking for the query

Each input is a dict with an optional text string and an images list of local image paths. Use text only, images only, or both for fused inputs. Queries and documents use different default instructions, which are part of the training format.

License

The Trident-specific contributions, including the fine-tuned weights, are released under the MIT License. The model is fine-tuned from Qwen3-VL-2B-Instruct, which is released under Apache-2.0, and the base model's license terms continue to apply to the weights derived from it. The training images come from third-party datasets (see Trident-Train) and remain subject to their original licenses.

Citation

If you find this work useful, please cite:

@article{trident2026chaos,
  title   = {Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers},
  author  = {Sun, Yubo and Peng, Chunyi and Yan, Yukun and Liu, Zhenghao and Xu, Zhipeng and Mei, Sen and Xin, Linlin and Zeng, Zheni and Sun, Maosong},
  journal = {arXiv preprint arXiv:2610.11816},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.11816}
}