TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models addresses a practical bottleneck in neural novel-view synthesis: models can produce high-quality images from multiple camera views, but rendering becomes slow and memory-intensive as the number of input views grows.
The problem: attention is accurate but expensive
Decoder-only large view synthesis models treat a scene as a learned latent state rather than explicitly reconstructing geometry. A model first processes context images and camera information, then generates a new view for a queried camera. This approach can handle difficult appearances such as reflections and translucent surfaces, but its cross-view attention must repeatedly consult a key-value cache containing the context tokens.
That cache grows with the number of context views. The authors explain that this makes rendering increasingly impractical at high resolution or with dense camera coverage. Test-time training methods such as LaCT replace attention with a fast-weight multilayer perceptron, or MLP, and achieve constant-complexity rendering. However, the paper reports that this speed comes with a substantial quality penalty.
TTB bakes attention into fast weights
The authors propose Test-Time Baking, or TTB, as a compromise between the two approaches. Instead of asking a fast-weight MLP to replace cross-view attention by learning directly from raw value tokens, TTB uses attention once during scene reconstruction and trains the MLP to imitate the resulting attention output.
In effect, the expensive global scene computation is shifted into a prefilling or reconstruction stage. The resulting fast weights become a compact scene representation that novel-view queries can use without consulting the full key-value cache. Rendering therefore has the same constant-complexity character as the TTT-style baseline, while the MLP has been trained from a richer target containing information aggregated across the context views.
This target is important. The paper’s ablations indicate that replacing the raw value-token target with cross-view attention output improves quality and broadens the contribution of views beyond the most dominant context image. The authors describe this as making the fast-weight MLP a collaborator with attention rather than a wholesale substitute for it.
Fixing a mismatch between context and novel views
TTB also addresses a less obvious problem in the reconstruction pipeline. Ordinary context tokens contain both camera information and image content, while novel-view tokens contain camera information only. If the same fast-weight MLP is updated using image-bearing inputs and later queried with camera-only inputs, it must generalize across mismatched modalities.
The proposed solution is a context pair. For each input view, TTB creates a ground-truth token carrying camera and RGB information, plus a rendering token carrying camera information only. The rendering token is used as the MLP input, while the corresponding ground-truth token supplies the cross-view-attention target. Because both resemble their roles during reconstruction and rendering, the authors argue that the MLP faces an easier mapping problem.
The method uses mean squared error for the inner-loop objective, performs multiple fast-weight updates, and learns a residual rather than the entire target. The paper reports that context pairing, residual learning, feedback to the ground-truth stream, and multi-step MSE optimization each contribute to the final result.
What the experiments show
On the DL3DV benchmark at 960p, the authors report 23 frames per second for TTB with 16, 32, or 64 input views. In their comparisons, this is up to 23 times faster than DVSM, the attention-based model it builds on, and 2.3 times faster than LaCT. With 64 input views, the reported TTB result reaches 30.20 PSNR, 0.897 SSIM, and 0.131 LPIPS, compared with DVSM’s 31.11 PSNR, 0.920 SSIM, and 0.104 LPIPS. The TTB variant using local duplicates reaches 31.33 PSNR, 0.919 SSIM, and 0.108 LPIPS at the same reported rendering rate.
These numbers illustrate the central tradeoff: standard TTB is faster at scene reconstruction than the local-duplicate variant, while duplication improves quality when many views must be compressed into the fast-weight state. Qualitative comparisons in the paper show that TTB preserves thin structures and fine details, including railings, window grilles, and small text, that the authors found visibly blurred by LaCT and LVSM.
The authors also evaluate zero-shot on MipNeRF360, Free, and Hike. They report that TTB generally approaches DVSM quality while rendering much faster, and outperforms the compared LaCT and LVSM systems on the reported settings. On Re10K with two input views, where the speed advantage is smaller, TTB is reported to reproduce DVSM-level quality.
Limitations and deployment tradeoffs
TTB does not eliminate computation; it relocates it. Its full attention-target version retains quadratic reconstruction cost in the number of context views, so it is less attractive when only one novel image is needed. The paper’s own end-to-end analysis says TTB overtakes DVSM after roughly five to seven rendered views, whereas the linear-target TTB variant leads earlier but gives up some quality.
The fast-weight MLP also has fixed capacity. As the number of context views grows, a single baked state can saturate. The authors mitigate this with a heuristic: cluster views by camera pose, bake a separate local copy for each group, and route each novel query to the nearest group. This preserves constant per-view rendering cost, but increases stored state and reconstruction work. The routing strategy is not learned, and the paper identifies learned capacity management as future work.
Why this matters in practice
TTB is most relevant to interactive viewers, virtual-tour systems, robotics or inspection workflows, and other applications that render many novel views from one captured scene. Its value grows when a scene is prefetched once and then explored through a long camera trajectory. A fixed-size baked representation can also reduce memory pressure compared with a key-value cache that expands linearly with context length.
More broadly, the authors suggest that baking could apply wherever attention provides strong context integration but its cache becomes too large: long-context language models, video systems, world models, and streaming perception. Those extensions are proposals rather than results demonstrated by this paper. The demonstrated contribution is narrower but useful: supervise a fast-weight module with attention’s outputs, align its input modality with deployment, and exchange one-time reconstruction cost for faster repeated rendering.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.