Why proxy video quality matters
Modern video-generation systems can use a rough, graybox video as a control signal. That proxy fixes important decisions such as scene layout, subject motion, camera placement, and shot changes, while the generator supplies a more realistic appearance. The advantage is controllability: instead of asking a model to invent an entire scene, a filmmaker or agent can specify its structure first.
The difficulty is creating the proxy. A coding agent must turn a script, photograph, opening frame, or reference video into executable Blender code, then render a time-varying scene. The authors of VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender argue that existing evaluations do not adequately test this combined problem. Many focus on text prompts, single objects, static scenes, or short clips, and some judge only the final video rather than the 4D scene that produced it.
A benchmark for executable 4D scenes
VibePrevisBench contains 242 tasks across four input types: 94 scripts, 39 photo tasks, 23 continuations, and 86 reference videos. The tasks cover seven scene domains and 33 places. Most are deliberately more demanding than a single-object reconstruction: they involve multiple people, several locations, moving cameras, interactions, timed events, and multiple shots. Outputs have a median duration of about 40 seconds.
Each task includes hidden ground-truth facts. These facts describe entities, relative layout, connected spaces, routes, events, camera behavior, cuts, and physical validity. For media-based tasks, they can also describe how the reconstructed picture compares with the reference. Agents receive the brief and optional media, then deliver a Blender program. The benchmark reruns that program in a fresh environment, rather than trusting the agent’s own render.
Two complementary checks
Rule-based Scene Check inspects the reconstructed 4D scene itself. It reads the exported objects, bounding boxes, cameras, and cuts frame by frame, then compares them with hidden facts in up to nine dimensions. This can catch an error that looks superficially acceptable on screen, such as filming a group from a high angle when the request specified an eye-level view.
VLM-based Screen Check asks whether the facts that passed the scene checks are actually visible and understandable in the proxy video. The judge considers recognition, action readability, visibility at key moments, and story comprehension. This second check matters because a technically correct action may be too small, occluded, or poorly framed for a viewer to recognize. The authors report that 31% of annotated actions passed the rules but were difficult or impossible to see clearly in the proxy video.
What the evaluation found
The authors evaluated 11 coding agents. GPT-6 Astra achieved the highest overall score at 77.7, followed by GPT-6.1 Sol at 76.1, Gemini 4 Argon at 75.0, and Opus 5.5 at 74.1. These scores average the scene and screen checks; they should not be read as a percentage of fully solved tasks. No agent came close to solving the whole benchmark: an agent reached a Rule-based Scene Check score of 90 on only 35 of the 242 tasks.
Performance diverged most when agents had to interpret visual input. The gap between the best and worst agent on the rule-based score was 12.0 points for scripts and 19.9 points for photos. The paper’s analysis attributes much of this difference to understanding space, picture structure, and camera motion rather than to writing Blender code. In one ablation, replacing video with a description derived from hidden ground-truth facts gave DeepSeek V4.1 Flash a 15.6-point overall improvement on five video tasks. The authors present this as evidence that perception is a major bottleneck, not as a complete estimate of what a better system would achieve.
The agents also shared systematic failures. When a reference used a low camera angle, 66% of shots were produced at eye level, while eye-level shots were wrong only 9% of the time. Only 4% of orbit moves remained orbits, and 47% became sideways moves. Half of video and continuation runs had more shots than the reference. These patterns suggest a tendency to fall back on familiar staging choices when a request is unusual.
More self-inspection did not reliably solve the problem. Agents viewed between 13 and 40 images per run, but the authors found little score improvement associated with viewing more images. In 1,377 of 2,582 runs with post-render comments, an agent said the result matched the reference or looked right; camera-angle errors were about as common in those runs as in others. The paper therefore recommends item-by-item verification against the brief, rather than relying on a general impression of visual quality.
Limitations and practical implications
VibePrevisBench is a useful diagnostic, but it is not a complete measure of cinematic or generative-video ability. The benchmark scores graybox scenes and proxy videos, not appearance, lighting, realism, or final production quality. The authors show two examples in which errors carried into a generated real-world video, but they explicitly leave correlation with real-world video quality for future work.
The study also evaluates each agent once per task, uses a fixed sandbox and time budget, and includes only 39 photo tasks. Some hidden ground truth for videos and continuations is estimated through reconstruction pipelines, while the screen score depends on a VLM judge. Judge agreement was strong at the agent level, but per-run agreement was lower. The authors also note possible model-family influence because some models helped construct or assess parts of the benchmark, although they report safeguards and no strong overall self-preference.
For practitioners, the message is concrete. A Blender-coding agent should represent camera height, lens, movement type, shot count, event timing, and object visibility as explicit constraints. It should render targeted checks for each requirement instead of merely reviewing a few attractive frames. Teams building agentic creative tools can use the benchmark’s split between scene correctness and screen legibility as a design pattern: verify what the program contains, then verify what a viewer can understand. In this setting, better code generation alone is unlikely to be enough. Reliable visual reasoning and structured verification are equally important.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.