WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents

WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents

Web agents are moving beyond simple webpage navigation. In practical settings, an agent may need to search a dashboard, update a CRM record, send a message, interpret a video, and coordinate information across several applications. Evaluating those capabilities is difficult: benchmarks must be broad enough to represent real software work, yet stable enough that two runs can be compared fairly.

WebArena-Pro, presented by Imene Kerboua and colleagues, is designed around that tension. The authors introduce 300 human-validated tasks across 20 self-hosted web applications and six domain categories. The benchmark includes productivity, communication, analytics, business operations, design and media, and travel workflows. It also treats audio and video as first-class inputs rather than limiting agents to text, screenshots, and webpage structure.

Why existing web-agent evaluations fall short

The authors identify three weaknesses in earlier evaluation setups. First, reproducible web benchmarks often cover only a small collection of applications. That can make it difficult to tell whether an agent has learned transferable interaction skills or has simply adapted to a narrow set of interfaces.

Second, most self-hosted benchmarks focus on text and vision. Real websites frequently embed podcasts, music, tutorials, and other temporal media. An agent that can read a page but cannot inspect the relevant audio or video is not fully equipped for those tasks.

Third, live websites offer ecological realism but are unstable evaluation targets. Pages, data, layouts, and policies can change between runs. WebArena-Pro instead uses seeded, containerized environments, allowing each task to start from a controlled state.

A broader environment with explicit media actions

Each WebArena-Pro application runs locally with fixed credentials and seeded data. Before a task begins, its container is reset to a baseline, so actions taken in one episode do not contaminate another. This design supports parallel evaluation and makes individual failures easier to reproduce.

The 300 tasks are divided into 204 single-site tasks and 96 cross-site tasks. The latter require coordination between applications, such as summarizing a support ticket in a knowledge-management tool. There are also 221 text-and-vision tasks and 79 audio-or-video tasks.

For media tasks, the benchmark extends the usual browser action space. Agents can call transcribe() for audio, extract_frames() for video, or use listen() and watch() when their model supports direct audio and video input. Transcripts are added to the accessibility-tree observation, while extracted frames are returned alongside the page representation.

Task creation combines human authorship with automated assistance. Annotators write the task intent and scoring rubric, an LLM-based triage stage flags apparently trivial or infeasible candidates, and human reviewers make the final acceptance decision. The authors emphasize that human validation is the release gate.

How success is measured

WebArena-Pro uses rubric-based evaluation rather than judging only whether an agent reached a final page. Rubrics can check actions, such as opening a record or applying a filter, and outcomes, such as whether a file was created or an answer contains the required information. Criteria are arranged in a dependency graph, and a task counts as solved only when every required rubric passes.

The verifier uses deterministic backend checks where possible, followed by heuristic checks and, when necessary, an LLM judge. This combination is intended to recognize valid solutions without relying exclusively on exact action sequences.

Results show substantial progress and substantial room to improve

The strongest reported overall success rate was 62.3%, achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 under a 50-action limit. OpenCode with Kimi 3.1 Pro followed at 60.7%. GenericAgent configurations performed lower, with Gemini 3.1 Pro in omni mode and Gemma4 31B each reaching 30.3%.

These results should not be read as a clean ranking of the underlying models. The paper evaluates combinations of models, agent harnesses, tools, and action spaces. The authors explicitly note that the comparison cannot isolate model capability from harness design. Nevertheless, the results demonstrate that even the best tested configuration failed to record success on a large fraction of tasks.

Cross-application coordination was consistently harder than single-site work. For example, Codex reached 67.2% on single-site tasks but 52.1% on cross-site tasks. Claude Code recorded the strongest cross-site result at 55.2%. This gap suggests that maintaining state, entities, and goals across application boundaries remains a major challenge.

Media access also mattered, although its effect depended on the agent and model. On a conservative subset of 43 tasks whose answers genuinely depended on media content, Gemini 3.1 Pro achieved 2.3% without media tools, compared with 34.9% using native audio and video access. Claude Code with Opus 5.5 rose from 14.0% without media tools to 48.8% with transcription and frame extraction. The authors therefore show that media tools can be decisive, while also finding that direct access does not guarantee correct temporal reasoning or visual counting.

Limitations and practical implications

WebArena-Pro does not yet test live voice interaction or streaming video. Its temporal capabilities are limited to prerecorded segments, transcripts, and sampled frames. The shared 50-step budget also does not equalize computation: agents differ in token use, observation frequency, latency, and native tool behavior. Reported costs likewise depend on provider pricing and caching.

For researchers, the benchmark offers a practical way to test transfer across interfaces, multimodal grounding, reproducibility, and cross-application planning in one environment. Teams building browser agents can use its failure patterns as an engineering checklist: verify final state changes, preserve exact constraints, inspect the requested media rather than relying on metadata, and reserve enough actions for submission.

For evaluators, the central lesson is that a high score on a narrow browser benchmark may conceal weaknesses in coordination and temporal media understanding. WebArena-Pro does not solve web-agent evaluation, but the authors provide a stronger stress test for systems intended to operate across the heterogeneous, multimodal web.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close