VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation

VibeVideoBench: Benchmarking Multimodal Agents for Code-Driven Video Generation

Recent advances in AI have enabled impressive video generation from text prompts using diffusion models. However, these pixel-based methods struggle to guarantee precise content such as exact formulas, charts, or synchronized narration. Addressing this gap, a new benchmark called VibeVideoBench evaluates multimodal AI agents that generate videos entirely by writing and executing code, producing every frame and sound programmatically.

The Challenge of Code-Driven Video Generation

Unlike traditional video generation models that synthesize pixels directly, code-driven video generation requires an agent to interpret a detailed specification, produce source code that renders visuals and audio, and deliver a fully playable video file. This approach enables exactness, editability, and synchronization of audiovisual elements, which are critical for applications like educational lessons, data stories, product demos, and explainer videos.

However, evaluating such agents is challenging because videos are complex multimodal artifacts without simple unit tests. Existing benchmarks focus on short, silent clips or static visuals, and do not assess the agent’s ability to produce temporally consistent, audio-visual outputs that meet detailed task requirements.

Introducing VibeVideoBench

The authors present VibeVideoBench, the first large-scale benchmark designed to assess general-purpose multimodal agents on open-domain, code-driven video generation with audio. Key features include:

  • 239 diverse tasks spanning 12 categories such as data stories, music-driven pieces, classroom lessons, paper explainers, and games, in English and Chinese.
  • Self-contained specifications that fix all factual details, timing, and audio policies while preserving creative freedom.
  • Strict evaluation protocol where agents run autonomously in identical sandboxes with a one-hour time budget, producing a finished MP4 video with sound and an editable source project.
  • Hierarchical, evidence-based evaluation combining automated program checks of delivery specs, about 13,000 task-specific rubric checks linked to requirements, and five quality dimensions (visual quality, aesthetics, content logic, timing, audio quality) scored by three independent multimodal judge panels.
  • Multimodal judges that route checks by modality to ensure audio checks reach only judges capable of hearing, enhancing reliability.

Empirical Study of Nine State-of-the-Art Agents

The benchmark was used to evaluate nine leading multimodal agent systems from six labs, including GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, and others. Each agent ran all 239 tasks fully autonomously in its native harness. The study produced 2,151 scored rollouts and over 6,500 video judgments.

Key Findings

  1. Benchmark remains challenging: The best systems, GPT-6 Astra and Claude Opus 5.5, satisfy every core requirement on only about 76% of tasks, indicating substantial room for improvement.
  2. Delivery reliability matters: Counting missing videos as zero quality changes rankings significantly. For example, Gemini 4 Argon delivers only 77% of videos and falls in rank when delivery failures are penalized.
  3. Time management is critical: Most failed deliveries result from final renders starting too late, running too slowly, or never starting. The best system finishes well before the deadline, while others risk losing videos due to poor planning.
  4. Self-inspection is common but not predictive: Nearly all agents inspect their own rendered frames, but the amount of self-inspection does not correlate with final video quality. Agents sometimes claim verification that does not hold.
  5. Judges align with human preferences at system level: The three judge panels produce system rankings that correlate strongly (Spearman ρ=0.98) with blind human pairwise preferences. However, agreement on individual video pairs is lower (~73%).

Evaluation Methodology in Detail

Each task includes a natural language prompt, optional input assets (e.g., 3D scans, sprites, lecture notes), a machine-readable delivery specification (resolution, frame rate, duration), and an audio policy specifying narration, music, sound effects, and ambience requirements.

Agents receive these inputs in a fresh sandbox with preinstalled tools like Python, Node.js, headless Chromium, ffmpeg, and a TTS engine. They have one hour to produce a complete MP4 video with sound, an editable source project, and a timing sheet.

Evaluation proceeds in three layers:

  • L1 Program checks: Automated verification of delivery specs, such as video resolution, frame rate, duration, presence of audio, and source project completeness.
  • L2 Rubric checks: Approximately 13,000 atomic pass/fail checks linked to specific task requirements, each tagged by modality (visual, audio, audio-visual). Judges provide timestamped evidence for each check.
  • L3 Quality scores: Judges rate five dimensions on a 1–5 scale with task-specific anchors, including visual quality, aesthetics, content logic, timing/motion, and audio quality/synchronization.

Insights on Agent Behavior and Cost

Agents employ diverse rendering pipelines, including vector graphics (cairo), rasterization (PIL/numpy), and browser-based 2D/3D canvases (three.js). Soundtracks are synthesized programmatically.

Despite universal self-inspection, the extent of frame viewing does not predict quality. Some agents finish rendering well before the deadline, while others start final rendering too late, causing failures.

API usage costs vary widely, from $0.32 to over $15 per task, with a moderate correlation between cost and quality. The authors identify a Pareto frontier balancing cost and performance.

Limitations and Broader Implications

The benchmark uses a fixed one-hour time budget per task, which may limit some agents’ ability to complete complex videos. The evaluation emphasizes adherence to specifications over creativity or originality. Judges, based on proprietary models, have limited sensitivity to fine audio-visual synchronization.

VibeVideoBench focuses mainly on 1080p resolution and English tasks, with some Chinese tasks included. Longer videos and other languages remain future directions.

Practically, this benchmark advances the evaluation of AI agents capable of producing precise, editable, and synchronized video content from code. This is valuable for education, marketing, product demos, and scientific communication. However, the ability to generate persuasive, precise-looking videos also raises concerns about misinformation and deceptive content.

Conclusion

The authors’ work establishes a rigorous, large-scale benchmark for multimodal agents generating code-driven videos with sound. Their extensive evaluation reveals that while top systems achieve impressive quality, reliable delivery and full adherence remain challenging. The benchmark’s hierarchical, evidence-based evaluation protocol and multimodal judge panels provide a robust framework for future research and development in this emerging paradigm.

All findings and results are attributed to the authors of the VibeVideoBench study. The benchmark and evaluation methodology represent a significant step toward assessing AI agents’ capabilities in complex, multimodal video generation tasks.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close