VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents

VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents

Multimodal agents are increasingly expected to search the open web, inspect images, compare candidates, and assemble evidence over many steps. Yet a system that can identify one object in a photograph may still fail when the task requires tracking several images, preserving a candidate set, or using a visual clue to decide what to search next.

VisSearch-Bench targets this less visible failure mode. The benchmark, introduced by the paper’s authors, evaluates what they call visual context management: organizing visual information, connecting it to object identities and external facts, and using those connections to guide later searches and the final answer.

What the benchmark measures

VisSearch-Bench contains 204 human-written tasks spanning seven primary domains and 25 subdomains. The tasks are organized into five families:

  • Set-wise Visual Search: inspect every item in a defined collection, then filter, compare, or summarize the results.
  • Sequential Visual Search: use findings from one search step to guide the next.
  • Hybrid Wide–Deep Search: combine broad coverage of a set with linked, multi-step investigation.
  • Competitive Visual Disambiguation: distinguish similar-looking objects using fine visual details.
  • Visual-Structure-Guided Search: use order, spatial position, or relationships in an image to determine the search path.

The questions are deliberately demanding. Examples include checking every image in a collection for a visual property, following a chain of objects selected from an image, and identifying a product or artifact whose local details separate it from near matches.

Each task includes a reference answer, traceable evidence, and task-specific scoring rules. The authors describe a human-led construction process supported by automated checks for source accessibility, field completeness, coverage, answer consistency, and possible text-based shortcuts. They also report a locked reference trajectory for each task, although systems are not required to follow that exact route.

Accuracy is not enough

The benchmark reports two main metrics. Accuracy is strict: a task counts as correct only when the complete answer contains all and only the required content. This is appropriate for list-building and structured research tasks, but it also means a mostly correct answer receives no full-task credit.

The second metric, Process Score, measures the correctness and completeness of individual results and subquestion answers. It helps distinguish a system that searched the right space but missed several entries from one that formed the wrong hypothesis at the beginning.

This distinction matters in practice. A research assistant may identify most relevant objects and retrieve useful facts while still failing to produce a complete final table. Conversely, a polished answer can conceal an incorrect visual match that contaminated every later step.

Results: search helps, but long-horizon visual reasoning remains difficult

The authors evaluate 16 multimodal models and agent configurations. Without external tools, no model exceeds 6.37% task accuracy. Adding search through the ReAct framework produces a substantial improvement in the reported paired comparisons, but the best complete-task result is still only 26.47%, achieved by ReAct with Kimi-K3. ReAct with GPT-5.5 reaches 25.49% accuracy and the highest overall Process Score, 49.52%.

These results show why the two metrics should be read together. Kimi-K3 leads on complete-task accuracy, while GPT-5.5 recovers more individual critical states on average. Neither result suggests that the benchmark is nearly solved: even the strongest reported system fully solves only about one-quarter of the tasks.

Performance also varies by task family. Set-wise search is particularly revealing. ReAct with GPT-5.5 reaches a 44.10% Process Score in that category, but only 8.82% accuracy. In other words, the agent often finds useful pieces without completing the required collection consistently. The authors also report strong category-specific results elsewhere, including 50.00% accuracy for GPT-5.5 on Visual-Structure-Guided Search and 44.12% accuracy for two systems on Competitive Visual Disambiguation.

Search frameworks are not interchangeable. In the paired experiments, ReAct outperforms DeerFlow for some backbones, while DeerFlow performs better for Qwen3.6-35B-A3B. The paper therefore does not support the idea that one agent scaffold is universally best.

Direct visual access matters

An ablation compares text-only retrieval, image captions, and direct access to retrieved images and crops. The full visual setting performs best for all three tested models. For DeepSeek-V4.1-Flash, for example, accuracy rises from 1.96% without tools to 22.55% with ReAct and full visual access. Captioning helps in some individual cases, but across the tested models it loses more complete tasks than it recovers compared with direct image inspection.

The authors also observe that visual tools are used most frequently early in a search, although they continue to appear later. That pattern is consistent with a search process in which initial images establish hypotheses and later visual checks validate or revise them. More tool calls, however, do not automatically produce higher accuracy.

Limitations and practical implications

VisSearch-Bench is a focused evaluation, not a complete measure of multimodal agent quality. Its tasks depend on publicly retrievable sources, and the authors’ curation choices determine which visual relationships and domains are represented. The strict all-or-nothing Accuracy metric is useful for testing completeness but can understate partial progress; Process Score partly addresses that limitation without replacing end-to-end success.

The reported experiments also use fixed tool budgets, a single Pass@1 run per task, specified search backends, image limits, and model-specific configurations. Results should therefore be interpreted as measurements under this protocol, not as universal rankings across every deployment environment.

For developers, the practical lesson is clear: adding browsing to a vision-language model is necessary but insufficient. Reliable systems need explicit candidate tracking, evidence provenance, visual reinspection, coverage checks, and consistency tests before finalizing an answer. For evaluators, the paper argues for benchmarks that score both final completeness and intermediate evidence recovery. And for users, a confident, well-formatted answer from a multimodal agent should not be mistaken for proof that it inspected every relevant image or connected every search step correctly.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close