Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

In the evolving landscape of multimodal AI, a key challenge is how to represent visual information in a way that integrates seamlessly with language models. Discrete visual tokens—analogous to words in text—offer a promising approach by encoding images as sequences of discrete symbols. This approach could unify vision and language under a shared modeling framework, simplifying architectures and training. However, a longstanding assumption is that discretizing continuous visual features inevitably causes semantic loss due to quantization errors, limiting the fidelity of discrete tokens compared to continuous representations.

The Problem: Semantic Fidelity of Discrete Visual Tokens

The core question addressed by the authors is whether discrete tokens can serve as semantically lossless representations of visual signals. Semantic loss here means losing meaningful information necessary for understanding the content of an image. Traditional wisdom suggests that quantizing continuous visual features into a finite set of discrete codes introduces errors that degrade semantic content, making continuous features an upper bound for performance in downstream tasks.

This assumption poses a fundamental bottleneck for building large-scale multimodal models that rely on discrete tokens to unify vision and language. If discrete tokens cannot faithfully represent visual semantics, their utility as a native interface for multimodal learning is limited.

Method: Revisiting Quantization and Introducing Factorized Vector Quantization (FVQ)

The authors challenge the notion that quantization error directly correlates with semantic loss. They use downstream multimodal understanding performance as a proxy for semantic fidelity, reasoning that a representation enabling accurate image-centric question answering must retain essential semantic information.

Key insights and methodological contributions include:

  • Quantization Error Is Not a Reliable Proxy: Experiments show that discrete representations with higher quantization errors can outperform those with lower errors in multimodal understanding tasks. This decouples reconstruction error from semantic fidelity.
  • Effective Bit Budget Governs Semantic Capacity: The authors propose that the semantic information retained by discrete tokens depends primarily on the effective bit budget—the number of bits used to encode each image patch. Increasing this budget generally improves downstream performance.
  • Limitations of Existing Multi-Codebook Quantizers: Common quantization schemes like Residual Vector Quantization (RVQ) and Product Quantization (PQ) saturate as more codebooks are added. This is because later codebooks encode progressively less informative residuals or narrower feature partitions, limiting semantic gains despite increased bit budgets.
  • Factorized Vector Quantization (FVQ): To overcome saturation, the authors introduce FVQ, which uses multiple symmetric parallel codebooks quantizing complementary, fixed-dimensional global views of the same visual feature. This design preserves the full feature information collectively before quantization, allowing the bit budget to scale effectively without diminishing returns.

FVQ employs a tight-frame constraint on projections to ensure that the multiple views collectively preserve the original feature’s geometry and energy. Each codebook operates independently and symmetrically, avoiding hierarchical residual encoding that weakens later codebooks.

Evidence and Results: Matching Continuous Representations and State-of-the-Art Performance

The authors conduct extensive experiments, including industrial-scale training on over 100 billion tokens of multimodal data. Their findings include:

  • Bit-Budget Scaling: Increasing the number of FVQ codebooks steadily improves multimodal understanding performance, ultimately matching the performance of continuous visual representations on average across eight challenging benchmarks.
  • Fine-Grained Visual Understanding: FVQ closes the gap in tasks requiring detailed visual information, such as optical character recognition (OCR), document parsing, and chart understanding. It outperforms prior discrete-representation models by large margins (22.8% over UniAR and 59.2% over Emu3-Chat) and is competitive with strong continuous-representation models like Qwen3-VL.
  • Generalization Across Vision Encoders: The bit-budget principle and FVQ’s benefits hold across different pretrained vision encoders, demonstrating broad applicability.
  • Qualitative Examples: The model successfully handles extreme-resolution documents, irregular handwritten answer extraction, and dense full-page OCR, illustrating its fine-grained semantic retention.

Limitations and Considerations

While FVQ shows promising results, some limitations and open questions remain:

  • Complexity and Training Cost: Industrial-scale training with 100B+ tokens and large models is resource-intensive, potentially limiting accessibility.
  • Codebook Optimization Challenges: Large codebooks can be difficult to optimize and may suffer from poor utilization or collapse, though FVQ mitigates this by fixing codebook size and increasing the number of codebooks.
  • Task-Specific Performance Variations: Although FVQ matches continuous representations on average, some tasks or fine-grained details may still benefit from continuous features.
  • Generality Beyond Benchmarks: The benchmarks used are comprehensive but may not capture all real-world visual understanding challenges.

Practical Implications

The authors’ work has several important practical implications for AI research and development:

  • Unified Multimodal Modeling: Demonstrating that discrete tokens can be semantically lossless supports the development of unified models that process vision and language natively under a shared token interface, simplifying architectures and enabling scalable training.
  • Efficient Scaling Strategies: FVQ provides a scalable quantization method that avoids saturation, enabling larger bit budgets and richer visual representations without sacrificing efficiency.
  • Improved Fine-Grained Visual Tasks: By overcoming information bottlenecks, discrete token models can now better handle OCR, document understanding, and chart reasoning, expanding their applicability.
  • Foundation for Future Research: The identification of effective bit budget as a key factor and the FVQ design principles offer a practical recipe for future discrete visual tokenizers and multimodal foundation models.

“These results support discrete tokens as a scalable interface for large-scale native multimodal foundation models.”

Conclusion

The authors provide a compelling reevaluation of discrete visual tokens, showing that semantic fidelity depends more on effective bit budget than on raw quantization error. Their novel Factorized Vector Quantization method overcomes saturation issues in multi-codebook quantizers and achieves state-of-the-art performance on diverse multimodal benchmarks. This work challenges prior assumptions about the limitations of discrete visual representations and establishes a strong foundation for building scalable, unified multimodal AI systems that treat vision and language as truly integrated modalities.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close