SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding
Speculative decoding can make large language models generate tokens faster by having a lightweight drafter propose several tokens before the larger target model verifies them. The target still determines the output, but it can process multiple candidates in parallel rather than decoding every token one at a time.
The authors of SpecPilot argue that released speculative-decoding systems often leave substantial performance on the table. Their central observation is that the best optimization depends on the deployment: the target model, drafter, GPU, numerical format, and whether the server handles one request or a batch. A technique that helps a single request may add too much per-row work under batching.
Why stock speculative decoding falls short
For a single request, reading the target model’s weights can dominate the decoding step. In that setting, verifying additional candidates may cost relatively little, yet a stock parallel drafter may verify only one candidate chain. Other likely target tokens produced by the drafter are ignored.
Batching changes the economics. Each request adds attention, recurrent-state, prefill, and verification work. As a result, the speedup over autoregressive decoding can fall sharply as batch size increases. The same wide tree or extra verification pass that is nearly free for one request can become a bottleneck for many requests.
This makes manual tuning difficult. The authors describe a deployment as a combination of serving regime, drafter, target model, and GPU platform, with each combination exposing different bottlenecks.
A ceiling-guided agentic loop
SpecPilot is an autonomous optimization process organized into repeated rounds. First, a cost model compares the current implementation with an estimated performance ceiling. The model separates potential gains from accepting more tokens, reducing kernel and memory cost, and removing overhead outside the profiled kernels.
Multiple coding agents then explore different directions in parallel. Successful patches are integrated, followed by targeted kernel tuning. A controller that the agents cannot modify measures changes in paired runs and promotes them only when the gain clears a statistical threshold. The system also tests promoted changes on unseen requests rather than relying only on development workloads.
Correctness is treated as a separate problem from task accuracy. SpecPilot’s token-forcing procedure feeds generated outputs back through a reference-style scoring path and checks individual token decisions against the stock system. This is designed to catch reward hacking: a system might appear faster and retain benchmark accuracy while quietly changing outputs.
Validated mechanisms are stored in structured memory with their code dependencies, evidence, limits, and correctness requirements. When the system moves to another model, drafter, or GPU, these mechanisms can be reused and retuned rather than counted as entirely new discoveries.
What SpecPilot discovered
- W12: a lossless 12-bit weight representation that reduces weight traffic while decoding weights inside the matrix-multiplication pipeline.
- Best-first tree verification: a way to use more of the candidates produced by parallel drafters, including for hybrid targets with recurrent state.
- Two-level verification: a lower-precision preliminary pass selects a promising chain, while only an exact target pass is allowed to accept tokens.
- Batching mechanisms: shared node budgets, adaptive per-request drafting blocks, and selective prompt admission to control verification cost.
W12 is particularly notable because the paper presents it as useful beyond speculative decoding. The authors report that it compresses dense BF16 checkpoints to about 12 bits per weight while preserving the original values exactly, and that its fused kernels outperform cuBLAS on the tested projection shapes. That claim is demonstrated on the evaluated models and hardware; it should not automatically be generalized to every GPU or workload.
Reported results
The evaluation covers four target models, three drafters, two GPU platforms, and five benchmarks. With one request, the authors report that SpecPilot raises the average speedup over autoregressive decoding from stock’s 4.46x to 10.87x in one RTX PRO 6000 setting. Across the reported single-request deployments, speedups over autoregressive decoding range from 8.86x to 10.87x.
Batching produces smaller but still substantial gains. For Qwen3.8-27B in FP8 on an H100, the reported speedup over autoregressive decoding rises from about 2x for stock systems to 3.90x or 3.93x, depending on the drafter. The paper also reports gains for smaller or weaker stock configurations, including Qwen3.5-4B and Gemma-4-12B with EAGLE-3.
The authors state that every sealed final passed token forcing and that no training or intentional quality reduction was used. In a direct single-request comparison on H100, SpecPilot is reported as 1.84x faster than the DDTree baseline.
Limitations and engineering costs
These results are not a universal guarantee for speculative decoding. The paper repeatedly shows that mechanisms are regime-dependent: wide trees and two-level verification can help one request but hurt under some batching conditions. The ceiling model is also an estimate, and its usefulness depends on accurate profiling, replay data, and implementation-specific kernels.
The optimization process itself is expensive. The reported rounds take hours and consume substantial GPU time, while agents create many code changes that are later rejected. Token forcing improves confidence in losslessness, but it is still a testing methodology rather than a formal proof that every deployment and input is divergence-free. The evaluation also focuses on selected models, GPUs, serving software, and benchmarks, so operators should reproduce the measurements on their own traffic.
Practical implications
For inference engineers, the main lesson is methodological: measure where the deployment loses time before choosing a decoding technique. A larger draft tree is not automatically better. Under single-request, memory-bound serving, lossless weight compression and more aggressive verification may be attractive. Under batching, schedulers should account for per-request row costs and hardware tile boundaries.
SpecPilot also suggests a practical role for coding agents in systems work. Agents can explore many implementation ideas quickly, but only when surrounded by independent timing controls, unseen-request validation, and output-level correctness checks. The authors’ contribution is therefore not simply a faster decoder; it is a framework for searching deployment-specific optimizations without treating speed as the only objective.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.