dNaViT: Discrete Codes are General-Purpose Visual Representations
Most vision systems still use different internal representations for different jobs. A semantic encoder may help a model recognize objects or answer questions, while a reconstruction-oriented latent space is better suited to preserving texture, typography, and other pixel-level details. The authors of dNaViT: Discrete Codes are General-Purpose Visual Representations ask whether one discrete vocabulary can support both needs—and also serve image understanding, generation, and editing.
The question matters because discrete tokens are the foundation of modern autoregressive language models. If images can be converted into useful discrete codes, vision and language models could share more of the same architectures, training objectives, and infrastructure. But images are continuous and information-dense. Reducing them to a finite vocabulary risks discarding exactly the small details needed for OCR, diagrams, faces, and faithful reconstruction.
How dNaViT works
dNaViT, short for Discrete Native-resolution Vision Transformer, starts with a frozen pretrained Qwen3-VL vision transformer. Rather than retraining that encoder, the system adds a lightweight Residual Modulator, or ResMod. This module combines features from multiple depths of the encoder and learns a spatially varying correction to the final semantic representation. Its residual path is initialized at zero, so the model begins with the original semantic features and gradually introduces reconstruction-relevant information.
The enriched features are then converted into discrete codes with grouped product quantization. Instead of using one enormous codebook, dNaViT divides the feature channels into groups and assigns each group its own codebook. Each spatial position therefore produces a tuple of code indices. This expands the combinatorial capacity of the representation without increasing the number of spatial positions in the image sequence.
The same codes feed two reconstruction branches. A semantic decoder is trained to recover the modulated visual features, using cosine alignment and mean squared error. A pixel decoder reconstructs the image using several losses, including pixel, perceptual, and texture-oriented objectives. The authors use a three-stage training schedule: continuous pixel pretraining, discrete semantic reconstruction, and joint optimization of the semantic and pixel objectives. A separate code-conditioned Refiner can replace the direct pixel decoder when sharper perceptual output is desired.
Evidence across understanding and generation
The paper evaluates dNaViT across reconstruction, classification, multimodal understanding, text-to-image generation, and image editing. For reconstruction, the 64-group version reaches 28.12 dB PSNR and 0.81 SSIM at 512 × 512 resolution, and 32.90 dB PSNR with 0.91 SSIM at 1024 × 1024. On TokBench at 1024 × 1024, it reports 63.08% text recognition accuracy, 78.27% normalized edit similarity, and 0.67 face similarity. The authors note that small text remains difficult and that stronger continuous systems such as FLUX still have an advantage.
Discrete codes also retain substantial semantic information. In linear-probe classification, dNaViT-G16 scores 79.47% on ImageNet-1K, only 0.93 percentage points below the corresponding continuous representation. Performance varies under distribution shift: the discrete version reaches 68.09% on ImageNet-R and 59.79% on ObjectNet, while the paper reports that UniTok remains stronger on some transfer benchmarks.
When paired with Qwen3-8B, the dNaViT-based model reports 70.7 on MMMU, 79.0 on MathVista, and 885 on OCRBench. On document and chart tasks, the authors say it substantially narrows the gap with continuous-representation models, matching Qwen3-VL-8B on DocVQA and OmniDocBench under the reported comparisons and leading the compared models on both CharXiv measures.
For generation, dNaViT-8B reports a 49.5 GenEval2 score, 52.17 on Qwen-Image-Bench, and 0.68 on SpatialGenEval. Its English and Chinese LongText-Bench scores are both 0.94. On GEdit-Bench, the reported overall score is 7.30. These results are strong among the open-weight baselines listed by the authors, but the paper also reports higher results for some proprietary systems.
What the results demonstrate
The central demonstrated result is not that discrete codes are universally superior. It is that, with enough capacity and carefully coordinated training, a single discrete representation can remain useful for tasks with conflicting information requirements. ResMod supplies detail without discarding the pretrained semantic anchor; grouped quantization supplies additional code capacity; and the dual objectives encourage the code to preserve both meaning and appearance.
The authors’ ablation evidence supports this interpretation. Compared with their SelfDistill baseline, ResMod maintains lower pixel reconstruction and semantic cosine losses during late training. They also report that continuous and discrete representations can produce comparable understanding performance in a controlled analysis, although the discrete version still trails on the reported average.
Limitations and practical implications
The comparisons require caution. Many baseline values come from published reports with different training data, image resolutions, sampling settings, model sizes, and evaluation protocols. The authors explicitly state that some reconstruction comparisons are not controlled or rate-matched. The G16 and G64 tokenizer variants also differ in training configuration and duration, so their gap does not isolate quantization group count.
The approach is not automatically lightweight. The tokenizer combines roughly 400 million frozen vision-transformer parameters with about 100 million parameters in ResMod, and the multimodal experiments use substantial training stages. The paper also says that discrete representations may need more data because their embeddings are learned from scratch. Finally, high-fidelity output can depend on a generative Refiner, which improves visual sharpness but adds another component and does not prove that every detail was perfectly preserved in the codes.
For practitioners, dNaViT points toward a unified interface for systems that need to read documents, reason over images, generate pictures, and edit existing inputs. Teams considering token-based multimodal models should treat the paper as evidence that discrete visual representations are viable—not as proof that they have replaced continuous latents. The most promising practical lesson is architectural: preserve a strong semantic backbone, add detail through a controlled residual path, and train the shared bottleneck against both semantic and pixel-level objectives.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.