Youtu-Parsing-Omni

News

• [TBD] Open-source evaluation code will be released.
• [2026-10] Technical report, model weights, vLLM plugin and inference examples released.

Introduction

Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, camera motion) and cognition (captions, narratives, reports). The output family is selected by the task prompt (--task in the examples, keys of prompts/youtu_parsing_omni.json).

Highlights (see the technical report for details):

• Unified schema – one JSON envelope for seven parsing families, driven by the task prompt.
• Omni encoder – image, audio and interleaved audio-visual video inputs (frames + audio track) in a single model.
• Strong results at a small size – state-of-the-art on OmniDocBench v1.6 (96.96 Overall), best open-weight model on OmniParsingBench (75.08 Avg., second only to Gemini-3-Pro), and competitive with specialized models on chemical-structure (ChemOCR) and music-score (PDMX-Synth) recognition (results).
• Easy to serve – a vLLM plugin, pinned serving settings, task prompts and inference examples are included.

Method

One encoder for every modality

Existing omni-modal parsers (a) encode pixels and audio with two separately pre-trained towers whose streams first meet inside the language model. Youtu-Parsing-Omni (b) replaces both towers with a single Youtu-Omni-Encoder, initialized from a pre-trained text language model to strengthen text reading. Thin modality-specific stems map pixels and log-mel frames to tokens; a shared bidirectional Transformer contextualizes an arbitrary packed mix of images, audio chunks and videos under one (t, h, w) positional encoding; modality-specific mergers and projectors then feed the tokens, together with the text prompt, to the Youtu-LLM decoder, which emits OmniSchema JSON. Only the stems and the merger / projector heads depend on the modality, so almost all perception parameters are shared.

Audio-visual fusion inside perception

The frames and audio chunks of a video are packed in temporal order. (a) A non-fusion layer gives every frame and every audio chunk its own attention window, while a few fusion layers start a new window at every audio-to-frame boundary, so a frame attends to the sound of the same instant. (b) Vision tokens take t from the frame index and (h, w) from the patch grid; audio tokens take t from their time offset with h = w = 0, so both modalities share one temporal coordinate system. Cross-modal alignment is thus computed inside the encoder rather than reconstructed by the decoder.

One schema for every task: OmniSchema-Aware On-Policy Distillation (OSAD)

All ten parsing tasks share OmniSchema, so after SFT and schema-routed RLVR a single self-evolving post-training procedure, OSAD, applies the same token-role-based objective across tasks. The student samples its own parses; rollouts that pass the admission gate (JSON validity, schema conformance, box syntax, repetition checks) are scored by a frozen copy of the model that additionally sees the reference parse. Because every output follows OmniSchema, each token is identified as content (Jensen–Shannon divergence) or structure (forward KL at a higher temperature), plus an all-token JS term. At every round boundary the student's LoRA adapters are merged and the student becomes the next teacher, so the parser and its supervision co-evolve without a hand-designed reward or a separately trained teacher.

Mode