Rethinking in Spikes: Mitigating Hallucinations in MDLMs with Step-Aware Decoding

Rethinking in Spikes: Mitigating Hallucinations in MDLMs with Step-Aware Decoding

Multimodal diffusion language models (MDLMs) generate text differently from conventional autoregressive systems. Rather than predicting one token at a time from left to right, they begin with a sequence of mask tokens and repeatedly predict and commit groups of tokens. This parallel, iterative process gives MDLMs strong global-context modeling capabilities, but it also creates a distinctive reliability problem: a small number of unstable denoising steps can introduce hallucinated details that survive into the final answer.

In the paper, Zhongxing Xu and colleagues investigate those unstable moments and propose StepRefine, a training-free decoding strategy that selectively replaces hard token commitment with temporary continuous refinement. The authors’ central claim is not that every denoising step is equally risky. Instead, they report that hallucinations are closely associated with local “entropy spikes”—sharp increases in predictive uncertainty that interrupt the generally downward trend during denoising.

Why denoising steps matter

At each MDLM step, the model predicts probability distributions for masked positions. Standard discrete decoding chooses the most likely token and then uses a remasking policy to decide which predictions become permanent. This is efficient, but it discards the rest of the probability distribution. When the model is uncertain, committing to one candidate may prevent later context from resolving the ambiguity.

The authors measure the mean Shannon entropy over masked positions and compare each step’s change with a local window of preceding changes. A step is flagged when its entropy deviation exceeds a local mean-plus-standard-deviation bound. Their analysis suggests that these spikes are rare, concentrated mainly in the early-to-middle portion of denoising, and associated with limited available context.

One analysis reported in the paper found that roughly 80% of hallucinated tokens fell within the identified entropy-spike region for the examined models. The authors also report that around 71% of spikes in LLaDA-V and 62% in Dream-VL occurred within the first half of denoising. These observations support the paper’s narrower conclusion: uncertainty dynamics can provide a useful signal for deciding when ordinary discrete decoding is least dependable.

How StepRefine works

StepRefine leaves ordinary denoising unchanged when the current step appears stable. At a detected entropy spike, it converts each masked position’s token distribution into a probability-weighted embedding. In effect, the model receives a soft representation of several candidate tokens rather than immediately receiving only the top prediction.

The method then performs additional forward passes within that same denoising step. Each pass updates the probability distribution and the corresponding soft embeddings, allowing competing candidate meanings to interact with the image, prompt, and surrounding tokens. Once the distribution becomes sufficiently stable—measured using average KL divergence between consecutive predictions—the controller stops refining. The model then performs normal discrete decoding and remasking.

This adaptive termination is important because refinement adds computation. The authors count those extra passes against the overall denoising budget rather than simply extending inference indefinitely. If a spike requires several refinement passes, later denoising indices are skipped so the total forward-pass budget remains fixed in their formulation.

What the experiments show

The authors evaluate StepRefine on four MDLM backbones: LLaDA-V-8B, LLaDA-O-8B, Dream-VL-7B, and LaViDA-L-8B. The benchmark suite covers general visual reasoning, hallucination, mathematical reasoning, and scientific reasoning. Across the reported main results, StepRefine improves performance over each corresponding base model, with average gains across backbones ranging from 2.3% to 2.8% according to the paper.

The reported improvements are also visible on hallucination-focused evaluations. For example, on LLaDA-V-8B, the MMHalu score rises from 3.35 to 3.50 and the Bingo score from 2.71 to 2.93. The method also improves several general reasoning results, including VStar, MMVP, and RealWorldQA, although the size of the gain varies by model and benchmark. Additional tables in the paper report positive changes across the tested mathematical and scientific tasks as well.

The authors further test compatibility with inference accelerators, including Fast-dLLM, dLLM-Cache, dKV-Cache, and SparseD. In the supplied results, adding StepRefine generally preserves the reported speedups while improving evaluation scores. This suggests that the approach targets a different part of the decoding process than cache reuse or sparse attention. It does not, however, mean refinement is free: the method still performs extra intra-step forward passes, even when the total budget is compensated elsewhere.

Important limitations

The evidence is encouraging but should be read within the paper’s scope. The evaluation covers four backbones and a substantial set of multimodal benchmarks, yet it does not establish that entropy spikes are a universal hallucination mechanism for every diffusion language model or task. The detection rule depends on a calibration-window size, and the ablation results show that very small windows can be noisy while very large windows can blur distinctions between denoising stages.

The method also cannot guarantee recovery of a correct answer. The authors include a qualitative failure case in which intra-step refinement does not resolve a difficult visual question. More broadly, a model can converge confidently on an incorrect interpretation; distributional stability is not the same as factual or visual correctness. Finally, the paper presents benchmark improvements rather than evidence from deployment settings, long-running production workloads, or user-facing reliability studies.

Practical implications

For teams experimenting with MDLMs, StepRefine offers a useful inference-time design pattern: monitor uncertainty over time, identify abnormal steps, and spend refinement selectively instead of treating every token or step identically. The approach is especially relevant when retraining is expensive or unavailable, because the authors present it as plug-and-play and training-free.

In practice, the paper points toward decoding systems that make uncertainty temporal rather than purely token-based. Still, implementers should measure latency, memory use, and quality under their own budgets, and should retain independent grounding or verification checks. StepRefine may reduce one source of hallucination, but the authors’ results do not justify treating it as a substitute for evaluation, retrieval, or human oversight in high-stakes applications.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close