Why diffusion VLMs keep rereading the same image
Diffusion vision-language models generate answers through repeated denoising steps. At each step, they predict some masked response tokens, update the partially completed answer, and repeat. The visual input normally remains fixed, yet the model processes its visual tokens again and again.
That repeated work is expensive because visual tokens dominate the input. In the LLaDA-V experiments reported by Donghyeon Cho, Joowon Kim, Seungho Shin, and Eunho Yang, visual tokens represented 89–97% of the model input across ten benchmarks. A forward pass without those tokens cost only 3–10% as much as a full multimodal pass.
The authors first asked a simple diagnostic question: does the image actually change the next-token decision at every denoising step? Comparing image-conditioned and text-only predictions along the same decoding trajectories, they found that the two top-1 predictions agreed on 83.4% of positions selected for unmasking. The image changed at least one selected decision on 22.8% of steps on average, ranging from 7.9% on MMBench to 43.8% on DocVQA.
LOWN: use the image as a conditional resource
The paper introduces LOWN, short for Look Only When Needed. Rather than pruning visual tokens permanently, LOWN decides at each step whether a full image-conditioned forward is necessary.
The method keeps an anchor: the latest prediction and confidence produced by a full forward. It then runs a much cheaper text-only forward, omitting the visual tokens while retaining the prompt and current response state. If the text-only prediction agrees with the anchor for a confidence-ordered prefix of masked positions, LOWN treats those tokens as certified. It can reuse the anchor predictions and unmask the certified tokens without reading the image again.
If agreement breaks before enough tokens are certified, LOWN runs a new full forward. That forward refreshes the anchor and prioritizes positions where the text-only and full predictions disagree. This combines two decisions in one rule: whether to consult the image and how many tokens to unmask.
The approach is training-free and does not require a tuned confidence threshold. Its key assumption is practical rather than absolute: agreement between a current text-only prediction and a recent image-conditioned prediction is evidence that the image is not needed for that particular update.
What the experiments show
On ten benchmarks covering visual question answering, mathematical reasoning, chart and document understanding, multiple-choice evaluation, and image captioning, the authors report that LOWN retained 99.6% of LLaDA-V’s baseline score while using 35.3% of its decoding FLOPs on average. When each benchmark was capped at 100% of baseline quality, the normalized result was 98.6%, a useful qualification because some individual LOWN scores exceeded the baseline.
LOWN achieved the highest score among the compared accelerated methods on eight of the ten benchmarks at matched compute. The paper also reports a 57.4% average reduction in measured generation time for standalone LOWN on the ten LLaDA-V tasks, with a maximum reduction of 73.2%.
The gains were not limited to LLaDA-V. On LaViDa, LOWN retained 99.7% of baseline quality at 17.4% of baseline decoding FLOPs on average. On VidLaDA-8B, evaluated under a controlled video protocol, it used 25.5% and 27.2% of baseline FLOPs on EgoSchema and MLVU respectively, with small score changes. These results suggest that the routing idea can transfer across related diffusion VLM architectures, although they do not establish equivalent benefits for all multimodal models.
Why routing complements pruning and caching
Visual token pruning and LOWN reduce different costs. Pruning processes fewer visual tokens whenever a full forward runs, but the removed information is unavailable later. LOWN leaves the complete visual representation intact and reduces how often the model consults it.
The authors found that combining LOWN with RedVTP at 50% visual-token retention achieved a normalized average score of 98.3% at 22.3% of LLaDA-V’s baseline FLOPs. LOWN also reduced the cost of other pruning configurations without retraining. The practical implication is that a system may be able to use less aggressive pruning while still reducing total computation.
LOWN also composes with Fast-dLLM, the key-value caching method evaluated in the paper. Across three reported tasks, adding LOWN reduced Fast-dLLM’s generation time by 38.1–50.1% and its FLOPs by 44.1–58.9%. However, the combined scores were lower than Fast-dLLM alone on those tasks, including a five-point decrease on LLaVA-Bench. Efficiency improvements therefore still require quality checks after composition.
Limitations and deployment questions
LOWN is not an exact certificate in the formal sense. A token accepted from the anchor can differ from a fresh full prediction, especially when the anchor becomes old. In the authors’ measurements, the fresh-full disagreement rate for certified tokens increased with anchor age, reaching 25.3% on DocVQA for an eight-step-old anchor. LOWN usually refreshes anchors quickly, and its measured acceptance error was 2.2–4.7% on text-only steps, but errors remain possible.
The method is also less effective when visual evidence remains necessary throughout decoding. Document and chart tasks required more full forwards than many question-answering benchmarks. LOWN’s compute is adaptive, so its cost cannot be fixed precisely before generation begins. The reported video evaluation additionally used a controlled protocol that differed from VidLaDA’s released evaluation setup, limiting direct conclusions about general video deployment.
Practical takeaway
For teams serving diffusion-based vision-language models, LOWN points to a third efficiency lever beyond shrinking the visual input or reducing the number of denoising steps: reduce the frequency of visual access. The method is especially attractive when visual tokens dominate inference cost and when the workload mixes image-grounded decisions with language-heavy continuation.
The broader lesson is that multimodal inference need not treat every decoding step as equally visual. Before adopting the technique, operators should benchmark agreement rates, certificate errors, latency, and task-specific quality on their own workloads. For fine-grained documents, charts, or other tasks where every step may depend on the image, adaptive routing may provide smaller savings. For suitable workloads, however, the authors’ results show that a model can look less often without permanently throwing visual information away.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.