On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
Large language models are capable but expensive to run, especially during autoregressive decoding. Smaller models are cheaper and faster, yet they often struggle with difficult reasoning, long instructions, and code generation. The paper’s central idea is to split those jobs: a large sender processes the prompt during prefilling, then transfers internal representations to a smaller receiver that generates the answer on its own.
This approach, called Prefill-to-Autoregressive Cross-Model Transfer, or PACT, could fit naturally into systems that already separate prefill and decoding. It also raises a basic scientific question: when the smaller model improves, is it actually using information from the larger model, or has the trainable connection simply adapted the receiver?
PLatour: a lightweight bridge between frozen models
The authors introduce PLatour, an open-source PACT framework designed to study that question at larger scales and across more model architectures. The sender and receiver remain frozen. Training updates only a set of communication bridges: zero-initialized, bias-free linear projections that map sender hidden states into the receiver’s residual stream.
These projections connect layers at corresponding relative depths. The projected sender states are added to the receiver during prompt processing, excluding the final prompt token. Once prefilling ends, the sender is no longer involved; the receiver decodes the response independently. Because PLatour transfers hidden states rather than directly replacing key-value caches, the authors report that it can also support hybrid-attention architectures.
The bridges are trained directly with reinforcement learning using task rewards and GRPO, rather than first aligning representations or distilling the sender’s output distribution. Zero initialization preserves the receiver’s original behavior at the start of training, while residual addition lets the receiver continue processing its own representations.
Strong benchmark results, especially for generation
Across the paper’s experiments, PLatour generally improves on the standalone receiver and on the open-source communication baselines tested by the authors. With a Qwen3-235B sender and Qwen3-1.7B receiver, PLatour reaches 69.29% on Chemistry generation, 89.35% on GSM8K, and 78.20% on MATH500. The corresponding receiver-alone scores are 41.43%, 80.86%, and 73.40%.
The authors also scale the method to a Qwen3.5-4B receiver. A 35B sender paired through PLatour scores 74.63% on AIME-135, 64.58% on HMMT, and 49.75% on IMO-AnswerBench, for a reported mathematics average of 63.32%. On coding benchmarks, it reaches 54.29% on LiveCodeBench v6, 94.97% on HumanEval, and 89.93% on MBPP. These results exceed the reported results for Mostik on shared benchmarks, despite Mostik using a 753B sender.
That comparison needs care. Mostik is closed-source, its training data are undisclosed, and the setups are not fully controlled. The authors also train separate PLatour bridges for mathematics and coding, so the results do not establish that one general-purpose bridge would deliver the same performance.
For efficiency, PLatour adds prefill work because both models process the input. However, the authors report that this fixed cost becomes less important as outputs grow longer. Total inference time approaches the receiver-alone baseline when decoding dominates, although the exact trade-off depends on hardware, prompt length, output length, and serving design.
Why benchmark gains are not enough
The paper’s most important contribution may be PACT-Check, a protocol for separating communication from adaptation. It includes four controls:
- Dummy Transfer replaces input-specific sender features with fixed template features.
- Irrelevant Transfer supplies representations from an unrelated question.
- Self-Communication uses the receiver as both sender and receiver, turning the bridge into a form of residual adapter.
- Trained Receiver fine-tunes the receiver directly with methods such as LoRA or full-parameter training.
The authors summarize this comparison with PACT-Score: the PACT system’s accuracy minus the strongest applicable control. A positive score is evidence that sender information contributed beyond the tested alternatives; a non-positive score means the improvement cannot be uniquely attributed to communication.
The results are strongly task-dependent. On Chemistry generation, PLatour’s gain remains above the controls, producing a positive PACT-Score of 3.34 percentage points in one reported setting. On GSM8K, however, the score is only 0.23 points, and on MATH500 it is negative. The authors therefore find evidence of useful sender information in some tasks, but not a general proof that PACT gains come from cross-model transfer.
This distinction matters because a bridge can behave like a parameter-efficient adapter or learned prefix. In some experiments, fixed or irrelevant signals preserve much of the improvement, while receiver-only training is an even stronger control. A system can therefore outperform the unmodified receiver without using question-specific knowledge from the sender.
Limitations and practical implications
The experiments focus mainly on related Qwen model families and task-specific training data. The authors identify open problems involving heterogeneous model families, representation mismatch, training stability, contamination control, and mechanistic understanding of what the bridge transmits. They also report instability in some large-sender runs, including a collapse during one MATH training run and a lasting performance drop after clipping spikes in a 397B-sender experiment. The reported observations do not establish clipping as the cause.
For practitioners, PLatour is best viewed as a promising research baseline rather than a drop-in replacement for receiver fine-tuning. It may be attractive when a larger model can improve prompt understanding while a smaller model handles long decoding, or when shared prefixes can be prefetched and cached. But evaluations should include sender-independent controls, not just receiver-alone comparisons.
The broader lesson is methodological. Cross-model systems need to demonstrate not only that performance improves, but that the improvement depends on information unavailable to the receiver through adaptation alone. PACT-Check offers one practical way to make that claim more testable. Until such checks become routine, impressive benchmark gains should be interpreted as evidence of a useful trained system—not automatically as evidence of successful latent communication.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.