VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
Physical AI systems often need to predict more than text. A robot may need a sequence of 2D and 3D waypoints, a bounding box, or the future motion of points in a scene. These visual plans give an action model an explicit spatial target, but conventional vision-language models usually generate their coordinates autoregressively, one token at a time.
The authors of VFlash argue that this is an awkward fit for structured visual outputs. The model already knows that a requested primitive is, for example, a six-token box or a trajectory with a defined number of points and coordinate fields. VFlash uses that known structure to decode an entire primitive in parallel.
The problem with token-by-token spatial planning
Autoregressive decoding is useful when every next token depends on the previous one. It is also expensive when a plan contains dozens or hundreds of coordinate tokens. Adding more waypoints, predicting both image and metric coordinates, or forecasting dense point motion all increases the number of sequential model passes.
That latency matters in physical systems, where a planner may need to update its prediction repeatedly as new camera observations arrive. Existing speculative decoding can propose several tokens at once, but its normal acceptance rule still requires agreement with the target model’s token sequence. A coordinate that is geometrically close but represented by a different token can therefore be rejected.
How Flash Primitive Decoding works
VFlash retains a frozen autoregressive target vision-language model and adds a lightweight drafter. The drafter receives features from the target and predicts masked positions in a complete visual primitive in one forward pass. A tool call specifies the requested length, while the primitive type specifies the layout: box corners, 2D and 3D trajectory segments, or dense motion fields.
This is more than simply increasing the speculative window. VFlash aligns decoding blocks with primitive boundaries and assigns each position a known role. A coordinate slot is not treated as an arbitrary next token; it is constrained to contain a legal coordinate. Delimiters, coordinate ranges, waypoint counts, and other schema rules can be checked without another neural evaluation.
The training objective also differs from ordinary speculative decoding. The drafter learns from annotated spatial primitives rather than merely trying to reproduce the target model’s continuation. At inference, a schema-valid visual primitive can be accepted in full, even when its token sequence does not exactly match the autoregressive target. Text remains on the conventional verified speculative-decoding path.
What the experiments show
Across the authors’ evaluated 2D robot trajectory tasks, VFlash reduced all reported errors relative to its autoregressive target. On the VABench-V benchmark, the paper reports roughly 40% lower errors than the best listed prior model, although the target used by the comparison was already stronger than several listed baselines.
The strongest results appear in 3D robot trajectory prediction. Across InternA1, AgiBot, and MolmoBot, VFlash improved average displacement error, final displacement error, and waypoint success rates for the tested target variants. In one AgiBot comparison, the fraction of waypoints within 5 cm increased from 50% with autoregressive decoding to 73%; with a joint 2D-and-3D target, it increased from 40% to 74%.
Bounding boxes illustrate an important trade-off. With an autoregressive fallback for uncertain boxes, VFlash approximately matched the baseline’s mean F1 while producing 2.6 to 2.8 times as many boxes per second. Directly accepting every schema-valid box was faster, but accuracy fell substantially on the more difficult Dense200 setting and was weaker at the strictest IoU threshold.
For dense 3D point motion, the authors report approximately 25 times faster single-rollout generation than their autoregressive target. Accuracy was mixed rather than uniformly better: VFlash improved several displacement measures, including DAVIS average and final displacement error, but lost ground on some point-wise and WorldTrack metrics.
Does faster planning help downstream control?
The paper evaluates whether the plans are useful to action models, not just accurate under geometric metrics. In MolmoSpaces, a world action model conditioned on VFlash’s plans reached 70.2% average success across four task suites, compared with 62.6% for the baseline. The largest gains were reported for Pick & Place and Close, tasks involving multiple dependent motions.
On the LIBERO family, visual plans helped most under the more challenging LIBERO-PRO perturbations. The paper also reports competitive autonomous-driving results on NAVSIM and Waymo after training separate action heads. These results show that VFlash plans can serve as conditioning features for downstream policies; they do not by themselves establish that VFlash is a complete robot or driving system.
Latency measurements on Jetson Thor, DGX Spark, and H100 hardware show the practical motivation. For MolmoSpaces, the authors report 5.1 to 5.2 times lower per-step latency on Thor and Spark, and 3.6 times lower on H100. The reported gains come primarily from decoding: a visual primitive that formerly required many target passes can be drafted in one pass and processed by the target once for context updates.
Limitations and practical implications
VFlash’s relaxed acceptance rule is a deliberate departure from lossless speculative decoding. It does not preserve the target model’s exact output distribution for visual primitives. The authors instead optimize for spatial accuracy and downstream usefulness, a sensible choice for coordinates but one that requires task-specific validation.
The method also depends on reliable schemas, coordinate representations, annotations, and fallback behavior. Direct box decoding was less robust than the guarded version, and dense motion results were not uniformly superior. The headline 25-fold speed figure is a preliminary single-rollout comparison on specific hardware and workloads, while broader latency results depend on the runtime and measurement boundaries described by the authors.
For practitioners, the central lesson is architectural: structured outputs should not automatically be decoded as unstructured text. If a planner knows the type, length, and role of every field, those constraints can become an inference interface. VFlash suggests that parallel visual planning is especially promising where latency matters, but deployment should pair direct decoding with validity checks, task-level evaluation, and conservative fallbacks.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.