Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Most multimodal systems can look at a document, image, audio clip, or video. Far fewer can turn all of those inputs into a consistent, grounded record that another model can inspect, search, and cite. The Youtu-Parsing-Omni Team presents its model as that missing perception layer: a 5-billion-parameter parser that converts heterogeneous media into structured outputs anchored in space or time.

The problem: multimodal inputs are hard to audit

Agents increasingly rely on information outside their language-model weights. They retrieve documents, inspect screens, call tools, and interpret recorded conversations or videos. Feeding raw pixels and waveforms directly into the agent can be expensive, and an answer produced from those inputs may not reveal which page region or time interval supports it.

The authors argue that parsing offers a cleaner interface. Instead of repeatedly asking an agent to reinterpret raw media, a parser can produce a structured record containing extracted content, locations, timestamps, and descriptions. The agent can then operate on that record and trace claims back to evidence.

Existing systems typically use different models and output formats for documents, speech, images, and video. That fragmentation becomes especially visible for symbolic material such as chemical structures, charts, geometry, and sheet music. Youtu-Parsing-Omni targets this gap with two unifying ideas: one encoder for every modality and one schema for every parsing task.

One shared perception encoder

Youtu-Omni-Encoder uses modality-specific input stems to convert image patches and log-mel audio features into tokens, but most of the Transformer computation is shared. The encoder uses bidirectional attention and a unified space-time positional representation. Visual tokens receive temporal, height, and width coordinates; audio tokens share the same temporal coordinate while leaving the spatial coordinates empty.

For audio-visual video, selected encoder layers allow frames and concurrent audio chunks to attend to one another inside short temporal windows. This is important because a useful video parse must connect what appears on screen with what is being said or heard at the same moment. In the authors’ design, some cross-modal alignment happens during perception rather than being left entirely to the language decoder.

The shared encoder is initialized from a pre-trained language model rather than trained entirely from scratch. In the reported ablation, text initialization substantially improved reading-oriented vision performance. At a matched 300-billion-token stage-1 budget, the initialized encoder reached an OCRBench score of 767.0 versus 636.4 for the scratch-initialized version. The same study found that the shared encoder could approach separate pre-trained vision and audio towers on a general multimodal average, although the comparison was conducted in a smaller controlled setting rather than as a full end-to-end replacement study.

One schema for ten tasks

OmniSchema represents image content as localized elements with normalized bounding boxes and represents audio or video content as timestamped segments. The fields can hold ordinary text, LaTeX, tables in OTSL, charts in Markdown, flowcharts in Mermaid, chemical structures as SMILES, and sheet music as ABC notation. Audio and video segments can include transcripts, speaker identifiers, acoustic events, visible objects, on-screen text, actions, camera motion, and event descriptions.

This common representation also supports the authors’ post-training method, OmniSchema-Aware On-Policy Distillation, or OSAD. After supervised fine-tuning and schema-routed reinforcement learning with verifiable rewards, the model generates its own parses. A frozen copy receives the reference parse as privileged context and supplies token-level targets on those same generated prefixes. The student sees only the original input and its own prefix.

OSAD treats JSON syntax and extracted content differently. Structure tokens such as keys and punctuation receive a forward KL objective, while content tokens use Jensen-Shannon divergence because multiple continuations may be valid. After each training round, the student becomes the next teacher. The team reports six OSAD rounds, with the parser and its supervision evolving together.

Results: strongest gains on temporal parsing

On OmniDocBench v1.6, the authors report an Overall score of 96.96, the highest among the compared systems. The gain over the team’s previous Youtu-Parsing model was concentrated in tables and formulas, while reading-order performance was also strongest in the comparison. On PureDocBench, the model reported an average score of 76.29 across clean, digitally degraded, and physically recaptured documents.

On OmniParsingBench, Youtu-Parsing-Omni achieved a 75.08 average and led the open-weight models evaluated by the authors. Its advantage was concentrated in temporal inputs: it scored 78.08 on audio, 74.18 on natural video, and 75.62 on text-rich video. The model was weaker on natural images and geometry than the strongest comparison systems in some dimensions. In audio, for example, its overall score benefited from localization and event-label performance even though its ASR score was lower than some baselines.

The reported domain-specific results are notable but should be read precisely. Chemical structure recognition reached a 54.2 exact fingerprint-match rate, compared with 49.6 for DeepSeek-OCR 2 in the cited comparison. On typeset sheet music, the model achieved a 22.73 character error rate, close to the specialist LEGATO result reported by the authors. These results demonstrate broad capability, not universal superiority across every metric.

Practical implications and limitations

For developers, the most useful idea may be the structured interface rather than the headline scores. A grounded JSON record could support document question answering, searchable meeting archives, video summarization, evidence-linked research tools, and agent workflows that need to distinguish what happened and when. A single schema also simplifies downstream ingestion: applications can handle elements and segments without maintaining an entirely separate contract for every input type.

However, the paper does not establish that one shared model is always cheaper or faster in production. The authors explicitly acknowledge that Youtu-Parsing-Omni lacks the inference acceleration used by its predecessor. It decodes end to end and autoregressively, so throughput may be lower on dense structured content such as tables. The OmniParsingBench comparison also uses an evaluation toolkit implemented by the authors, with schema repair and exclusions for unavailable media and an annotation-problematic camera-video subset. Those choices make the reported comparison useful, but they limit direct comparability with numbers reported elsewhere.

The practical conclusion is therefore measured: Youtu-Parsing-Omni shows that a compact open-weight checkpoint can combine document-grade parsing with audio and video grounding in one system. Its strongest evidence is in structured documents and temporal media. Teams considering it should validate latency, schema reliability, transcription quality, and performance on their own data before treating it as a universal replacement for specialist pipelines.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close