MMFlash: Faster Speculative Decoding for Vision-Language Models

MMFlash: Faster Speculative Decoding for Vision-Language Models

Vision-language models can answer questions about images, charts, documents, and videos, but generating their answers remains expensive. Like other autoregressive language models, they produce output one token at a time. This becomes a significant bottleneck when a model generates long reasoning traces or describes lengthy video content.

The authors of MMFlash: Faster Speculative Decoding for Vision-Language Models target this bottleneck with a diffusion-based speculative decoder designed for multimodal systems. Their method proposes multiple tokens in parallel, then asks the larger target model to verify them. Because verification preserves the target model’s decoding policy, the claimed acceleration is lossless: accepted tokens do not change the output distribution of the target model.

Why visual tokens are difficult to draft

Speculative decoding works best when a lightweight drafter makes predictions that closely match the target model. Existing diffusion drafters such as DFlash can propose a whole block of tokens in one forward pass, avoiding the repeated sequential passes required by conventional autoregressive drafters. However, applying this approach to vision-language models introduces a problem: some output tokens depend heavily on visual evidence, while others are mostly predictable from the text prefix.

To measure that distinction, the authors define a token-level visual dependence score. They run the target model twice on the same prefix: once with the image present and once without it. The score is the KL divergence between the two next-token distributions. A high value means that visual input substantially changes what the target model expects to generate.

Their analysis identifies a clear weakness in standard diffusion drafting. Across the reported study, token acceptance falls from 46% in the least visually dependent quartile to 22% in the most visually dependent quartile. The most visually dependent quartile also contains 44% of first rejection points, compared with 7% for the least dependent quartile. Training a DFlash drafter on multimodal data does not eliminate this pattern.

How MMFlash changes drafter training

MMFlash turns that diagnostic into a training signal. Instead of weighting every position only according to its location within a draft block, it increases the loss for tokens that are both visually grounded and still difficult for the drafter to predict.

The method first converts visual-dependence scores into corpus-wide quantiles. It then combines this value with the target model’s confidence. The resulting emphasis is highest when visual input changes the prediction and the target model has a relatively definite answer. This is intended to avoid spending excessive capacity on visually sensitive positions where the target itself is uncertain.

The weighting also accounts for the mechanics of verification. Since verification accepts a prefix of a draft block, errors early in the block prevent later tokens from being useful. MMFlash estimates the probability that each position will be reached and uses that estimate in the loss. A further dynamic factor gives more weight to tokens the drafter currently misses, allowing the extra emphasis to fade as the prediction improves.

For long videos, the authors add sparse attention. Their measurements show that the drafter’s attention is concentrated near the question and recently generated answer, even when the target context contains more than 40,000 tokens. The proposed pattern retains a few initial attention-sink tokens, strided visual tokens, all text tokens, and a local window. This reduces the amount of context processed by the drafter while preserving selected visual information.

Reported results

The authors evaluate MMFlash primarily with Qwen3.5-4B and Qwen3.5-9B across seven image benchmarks covering chart and document understanding, general visual question answering, and STEM reasoning. They compare it with native multi-token prediction, EAGLE-3, and DFlash.

Under greedy decoding, MMFlash reaches average speedups of 3.00–3.05 times across the two target models and benchmark groups. On the reported image tasks, it improves on DFlash by 6.7–10.9% on average and records the highest average acceptance lengths among the compared methods. At temperature 1, the reported average speedups range from 2.54 to 2.70 times.

The video results show a more incremental but still meaningful contribution. Sparse attention alone improves acceptance length and speed in the authors’ ablation, while additional video fine-tuning produces larger gains. Across LongVideoBench, MVBench, and Video-MME, the final configuration reports average speedups of 2.33–2.71 times, outperforming DFlash by 11.2–23.2% in the presented experiments.

MMFlash also remains faster than the baseline as serving concurrency rises from 1 to 32 in the SGLang tests. The advantage narrows as serving becomes more compute-bound, but the authors report that throughput remains higher than baseline in the tested settings. Additional experiments on vLLM, Gemma4-E4B-it, and Qwen3.8-27B are presented as evidence that the approach is not limited to one model family or scale.

Limitations and practical implications

The results are promising, but they do not establish that MMFlash will deliver the same gains in every deployment. The main evaluations use selected benchmark samples, specific target models, fixed inference configurations, and a particular serving stack. Real-world performance will depend on hardware, batching, memory movement, target-model size, output length, and the relative cost of the drafter.

Training also requires target-model signals. The authors report using online training to avoid the very large storage requirements of precomputing hidden states, but this shifts cost into the training process. The visual-dependence score itself requires running the target with and without visual input when preparing the training signal. That cost is paid offline or during training rather than at inference, but it remains relevant for teams building custom drafters.

Long-video drafting remains the less mature part of the proposal. The sparse pattern is deliberately simple and can discard information needed for a particular question. The authors identify question-dependent visual selection, frame-level dependence estimates, and redundancy reduction as future directions rather than demonstrated capabilities.

For practitioners, MMFlash suggests a useful design principle: multimodal acceleration should prioritize the tokens that are hardest to predict because of visual evidence, rather than treating all output positions equally. Teams already using DFlash-style speculative decoding could view visually aware loss weighting and sparse video attention as practical extensions to test. The paper’s evidence supports those ideas on its evaluated models and benchmarks; broader production benefits will require independent profiling and validation.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close