Flide: Accelerating Decision Inference with Exact State Forking
LLM serving systems are usually designed to generate text one sequence at a time. That design becomes inefficient when a model is used as a decision engine: given a long context, the system may need to answer many short, categorical questions about the same state. Routing, evaluation, agent control, and other latency-sensitive tasks often need probabilities over options rather than a paragraph of generated text.
The authors of Flide: Accelerating Decision Inference with Exact State Forking identify a conflict in current inference engines. They either process each question as an independent sequence, repeating much of the shared context, or concatenate questions into one sequence with delimiter tokens. The first approach wastes computation; the second can change the model’s hidden states and therefore alter the decision probabilities. Flide is designed to share computation without changing the logical input seen by any question.
The problem with treating decisions as generation
Logically, a decision request is a tree. A shared state forms the trunk, and each question is a separate branch that should see the trunk and its own preceding tokens, but not sibling questions. Physically, however, existing engines tend to represent each branch as a separate sequence. Prefix caching reduces some repeated work, but the paper argues that it remains imperfect for hybrid models containing recurrent layers, where states may only be restored at coarse checkpoints.
Multi-item scoring takes the opposite approach by placing multiple questions in one stream. According to the authors, delimiter tokens inserted between items become part of the computation and can shift the resulting distributions. The paper also highlights numerical nondeterminism: changing the batch or server load can change the order of floating-point reductions, potentially changing a thresholded decision even when the request is identical.
How Flide works
Flide, short for Flat-Graph Inference for Decision Models, separates a request’s logical tree from its physical GPU layout. It flattens the states and question branches into one token stream, while GPU-resident metadata records request IDs, branch IDs, positions, segment boundaries, and dependencies. The stream is therefore compact, but the metadata preserves the tree structure needed for correct execution.
Its central mechanism, ForkAttention, handles both ordinary attention and Gated DeltaNet recurrent layers. In full-attention layers, a question can attend to its own shared state and earlier question tokens, while sibling branches remain invisible. For recurrent layers, masking alone is insufficient because the recurrent state already summarizes earlier tokens. Flide first processes the shared state segments, then starts each question from the corresponding recurrent state and convolution window. The authors prove that, in exact arithmetic, this produces the same hidden states as processing every question independently.
The engine also uses captured CUDA graphs whose shapes depend mainly on the flattened stream’s capacity rather than the exact mixture of requests. Request-specific structure is supplied as device data before replay, allowing different rounds to reuse a small set of graphs. Selective readout avoids unnecessary final-layer work because only the last token of each question is needed for the option probabilities.
For agent histories that grow over time, Flide can save internal checkpoints. Later requests reuse the unchanged prefix state and recompute only a mutable tail and newly added tokens before forking questions. An optional batch-invariant mode fixes reduction orders so that answers remain bitwise identical across different loads on the same hardware and software configuration.
What the evaluation found
The authors evaluated Flide on public decision and reading-comprehension workloads, including RACE, QuALITY, MultiRC, and Decision Index, using Qwen3.5 and Gemma 4 models ranging from small models to 31B variants. They compared it with vLLM, SGLang, SGLang’s multi-item scoring mode, and dedicated decision-serving systems for Open-Jev.
The headline result is that Flide achieved up to 9.5 times lower latency and up to 8.5 times higher throughput than the strongest general baselines in the reported multi-question settings. The gains were largest when many questions shared a long state, because Flide computed that state once rather than repeating or partially restoring it for each branch. On one state with 64 questions, the authors report a 12.7-times speed advantage over vLLM in one analysis.
Speed did not come at the cost of the measured answer fidelity. Across the reported Qwen3.5 workloads, Flide’s top answers were closest to a float32 reference in total variation distance. Its load-related answer instability reached at most 0.2 percent in the cited tests, while vLLM reached up to 3.2 percent and SGLang up to 1.7 percent. The paper reports that delimiter-based multi-item scoring shifted distributions substantially more and flipped up to 29 percent of categorical decisions in its comparison, versus a maximum of 2.3 percent for Flide.
Ablations support the design. Removing state forks caused particularly large slowdowns for long, multi-question inputs; removing CUDA graphs hurt short requests; and removing selective readout reduced throughput. On growing agent histories, the authors report that Flide’s latency increased by roughly one millisecond per additional thousand history tokens in the tested setup, and that it completed 4.4 to 7.2 times as many agent steps per second as the strongest baseline.
Limitations and practical implications
These results are promising, but they are not a universal benchmark of LLM serving. The experiments were conducted on one NVIDIA H200 and on the models, workloads, software versions, and prompt formats described by the authors. The paper’s exactness guarantee is stated for exact arithmetic; real implementations still operate in floating point, even though the reported errors were near rounding levels. The bitwise-invariant mode also trades some throughput for reproducibility, with the paper reporting a cost of up to 10 percent at 64 clients in its tests.
Flide is most relevant when one expensive context supports multiple short decisions, or when an agent repeatedly extends a state and asks several questions at each step. For single-question workloads dominated by model computation, the advantage can be smaller. Deployment teams should therefore measure their own question-to-state ratios, model architectures, memory limits, and determinism requirements rather than assuming the maximum reported speedup.
The broader contribution is architectural: decision inference should be represented as shared computation plus isolated branches, not as unrelated generation sequences. If that abstraction survives beyond the tested models and hardware, Flide’s state forking and graph-based execution could offer a useful foundation for faster, more reproducible routing and agent-control systems.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.