Multi-Token Prediction Done Right

Multi-Token Prediction Done Right

Next-token prediction is the standard training objective for language models, but many recent systems also use multi-token prediction (MTP). Instead of predicting only the next token, an MTP model learns to predict several future tokens at once. This can improve downstream capabilities and provide draft tokens for speculative decoding, a technique that accelerates generation by allowing a model to verify several proposed tokens in parallel.

In Multi-Token Prediction Done Right, Jayden Teoh of MIT argues that the usual MTP objective contains a useful learning signal alongside a harmful one. The paper introduces Soft Multi-Token Prediction, or Soft-MTP, which attempts to preserve the useful signal while removing the noisy component. The authors report improvements in language-model loss, zero-shot evaluation, speculative decoding, and post-training math and code performance.

The problem with standard MTP

MTP heads and the ordinary next-token head typically share a linear output layer. For a given future token, the MTP cross-entropy gradient can therefore be decomposed into two exact parts. The first is a KL self-distillation gradient: it encourages an MTP head to match the model’s own next-token probability distribution at the corresponding position. This is directly relevant to speculative decoding, where draft tokens must agree with the model’s verifier.

The second component is a residual gradient that resembles ordinary cross-entropy supervision against the sampled training token. Teoh’s analysis identifies this residual as a source of variance. It is not useless in an absolute sense, but the paper argues that it has a lower signal-to-noise ratio than the self-distillation component and can interfere with both MTP and next-token learning.

The proposed explanation is a bias-variance tradeoff. The residual includes irreducible noise from using a one-hot target. The self-distillation term, by contrast, is determined by the model’s current predictions. Soft-MTP deliberately biases MTP heads toward the model’s next-token distribution. That bias may be desirable when the main operational goal is to produce drafts that the same model will accept during speculative decoding.

How Soft-MTP works

A straightforward solution would explicitly minimize the KL divergence between the next-token and MTP distributions. However, language-model vocabularies are large, and explicitly materializing both distributions can create memory and kernel-compatibility problems. The paper’s central technique, called KL gradient recovery, avoids this cost.

The method uses straight-through gradient routing. In the forward pass, the relevant NTP hidden state is numerically unchanged. During backpropagation, however, the computation routes a negative copy of the NTP cross-entropy gradient toward the MTP hidden state. This cancels the residual portion of the MTP gradient, leaving the KL self-distillation gradient for the MTP heads and shared trunk.

The output head is treated differently. Soft-MTP continues to train the shared output layer with cross-entropy against the data, rather than with the recovered KL gradient. The authors argue that this prevents the output layer from taking shortcuts that reduce the KL objective without making the MTP representations genuinely better. In the supplied pseudocode, the change is presented as a one-line modification to the hidden-state path, with no new parameters.

What the experiments show

The authors pre-trained LLaMA-style models ranging from 167 million to 1.2 billion parameters. Their main experiments used DeepSeek-style sequential MTP with three additional prediction heads, 31.5 billion FineWeb-Edu tokens, and a fixed MTP loss coefficient. They also report a result with Meta-style MTP in an appendix.

The gradient analysis supports the paper’s diagnosis. Across the tested scales, the residual had roughly four times lower signal-to-noise ratio than the KL component. At the same time, the residual accounted for approximately 80% of the MTP gradient reaching the shared trunk. The gap in signal-to-noise ratio widened from the smallest to the largest tested model. A separate diagnostic found that a step along the KL component reduced a proxy for next-token loss, while a step along the residual increased it.

Soft-MTP outperformed standard MTP on the paper’s main comparisons. It achieved lower next-token perplexity at every tested scale and lower MTP loss from 689 million parameters onward. In self-speculative decoding, it produced MTP distributions closer to the model’s own verifier, with better acceptance and speedup results across FineWeb-Edu, Wikipedia, Books, Science, and Math. The reported average gains over standard MTP were modest, roughly 1.8 to 2.2 percent in accepted-token length and 1.4 to 1.5 percent in speedup.

Zero-shot accuracy also favored Soft-MTP over standard MTP at each scale, with the advantage becoming more noticeable at 689 million and 1.2 billion parameters. After NTP-only fine-tuning, the 1.2B Soft-MTP model led standard MTP on seven of eight reported math and code metrics. These are promising results, but they remain results from the authors’ controlled experiments rather than evidence that every MTP implementation will benefit equally.

Efficiency and limitations

At 1.2B parameters, the authors report that Soft-MTP matched standard MTP in measured FLOPs, fused-cross-entropy memory, model-flop utilization, and nearly identical throughput. Direct KL-MTP was more expensive in their comparison because it materialized dense vocabulary distributions. Soft-MTP also matched or exceeded direct KL-MTP on most metrics, although direct KL-MTP performed better on the reported math fine-tuning results.

Several limitations matter for interpretation. The main scaling study stops at 1.2 billion parameters and uses one primary corpus, one model family, a particular optimizer, and a fixed training budget. The paper does not establish behavior at frontier scale, under different data mixtures, or with every MTP architecture and parallelism strategy. Its proxy analyses characterize gradient directions but do not prove that those directions explain all final-model gains. Finally, the claimed lack of overhead is demonstrated in the reported implementation and hardware setup; production systems may encounter different communication, compiler, or sharding constraints.

Practical implications

For teams already training MTP models with a shared linear output head, Soft-MTP is an unusually low-risk optimization to investigate. It does not require extra parameters or explicit vocabulary-sized KL tensors, and it is designed to preserve fused cross-entropy kernels. The most immediate potential benefit is better self-speculative decoding, with downstream quality improvements as a secondary incentive.

The broader lesson is methodological: an objective’s gradient may contain several competing signals, and changing gradient routing can be more useful than adding a new loss. Whether Soft-MTP becomes a standard training recipe will depend on replication at larger scales and across more architectures. The paper provides a compelling case to run that experiment.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close