DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models
Reinforcement learning has helped language models improve at tasks such as mathematical reasoning, but applying it to diffusion language models introduces a distinctive challenge: the model does not generate a response strictly from left to right. Instead, it repeatedly denoises a partially masked sequence, committing some tokens while refining others. A final reward can indicate whether the answer is correct, but it says little about which commitments helped or harmed the later denoising process.
The authors from Zhejiang University, the National University of Singapore, and SAP propose Denoising Token Credit Assignment, or DTCA, to address this problem. Their framework estimates how individual token commitments affect the model’s confidence in the remaining masked positions. It then uses those estimates to select useful denoising steps, distribute reinforcement-learning credit among tokens, and calibrate updates according to predicted entropy changes.
Why diffusion changes credit assignment
In an autoregressive model, each generated token naturally defines a step in the trajectory. In a diffusion language model, several tokens may be committed at the same denoising step, and each commitment can change the predictions for many positions that remain masked. The interactions are therefore less localized. Directly measuring every token’s effect during training would also require extra model evaluations and could produce noisy estimates that vary across rollouts.
DTCA addresses this instability by grouping tokens according to part of speech. In preliminary analyses on GSM8K rollouts, the authors report that categories such as nouns, verbs, numerals, and punctuation exhibit different commitment patterns and different effects on the confidence of remaining masked positions. Verbs showed the largest average confidence gain in the reported analysis, while determiners and adpositions showed smaller gains. These observations motivate POS-level aggregation as a practical approximation to token-level contribution.
How DTCA works
For an individual commitment, DTCA compares the model’s aggregate prediction confidence on positions that remain masked before and after the commitment. The contribution estimate is the positive part of that confidence improvement. A commitment receives a larger score when it appears to make the remaining denoising problem easier for the model.
The authors first calculate these estimates from offline rollouts rather than recomputing them during every reinforcement-learning update. Estimates are averaged within POS categories and transformed into bounded weights from 0.8 to 1.2. The bounded range is intended to preserve the basic reinforcement-learning objective while still emphasizing categories with stronger estimated refinement effects.
Those weights serve two purposes. First, they influence timestep selection. DTCA scores a denoising step using the contribution weights of its committed tokens and their uncertainty, assigning priority to steps containing relatively low-confidence commitments with potentially meaningful downstream effects. The method selects the highest-scoring steps until reaching a specified share of the total priority mass, subject to lower and upper limits.
Second, the weights determine token-level credit. Within a selected step, tokens from higher-contribution categories receive a larger share of the response-level advantage, while normalization keeps the average credit for that step unchanged. This does not create a new reward for each token; it redistributes the existing reinforcement-learning signal using information from denoising behavior.
DTCA adds an entropy-based calibration stage as well. The method estimates whether each policy update is likely to increase or decrease token-level predictive entropy, then balances the two types of updates within a response. The authors describe this as an adaptation of entropy-flow calibration to diffusion denoising states. They report that this component is intended to make later policy updates more stable rather than simply force entropy downward.
Reported results
The experiments start reinforcement-learning training from LLaDA-8B-Instruct without an intermediate supervised fine-tuning stage. Training uses mathematical reasoning data, while evaluation measures final-answer accuracy on MATH500 and GSM8K at generation budgets of 128, 256, and 512 tokens.
According to the authors, DTCA improves over the base model at every reported budget. On MATH500, the gains are 9.0, 6.8, and 4.8 percentage points at budgets of 128, 256, and 512, respectively. On GSM8K, the gains are 5.3, 2.1, and 2.5 points. The strongest improvements occur at the smallest budget, suggesting that the approach may be particularly useful when denoising computation is constrained.
The ablations support contributions from both major mechanisms. Removing contribution-based credit lowers the full method’s 256-budget scores by 3.8 points on MATH500 and 2.4 points on GSM8K. Removing entropy calibration causes smaller drops at that budget, but the effect is larger at 512 tokens: 4.2 points on MATH500 and 3.5 points on GSM8K. Training analyses also associate contribution-based credit with different gradient behavior and entropy calibration with slower late-stage KL divergence growth.
These results demonstrate an improvement for the reported LLaDA-8B-Instruct mathematical-reasoning setup. They do not, by themselves, establish that POS categories are the best credit-assignment units for every diffusion language model or task.
Limitations and practical implications
The method depends on POS tagging and aligning word-level annotations with model-token positions. Errors or ambiguities in that alignment could affect the fixed contribution table. The contribution definition is also heuristic: improved confidence on remaining masked positions is a useful observable, but it is not a formal causal measurement of whether a token ultimately produces a correct answer.
The evidence is concentrated on one base model and mathematical reasoning benchmarks. The authors explicitly identify code generation, other tasks, and other diffusion architectures as areas requiring further evaluation. The reported entropy changes are often first-order predictions rather than direct measurements after a parameter update, and the POS analyses compare training trends across runs rather than matched tokens in identical states.
For practitioners experimenting with reinforcement learning for diffusion language models, DTCA offers a concrete design pattern: use denoising trajectories as a source of process information instead of applying one undifferentiated outcome advantage everywhere. Offline analysis can identify which token groups and denoising stages carry useful signals, while bounded reweighting and entropy monitoring can limit destabilizing updates. The broader lesson is that diffusion models may need RL objectives designed around their iterative refinement process, not simply adaptations of autoregressive training recipes.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.