Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning
Multimodal large language models can often recognize what appears in an image but still fail when they must reason across several visual elements and multiple steps. A model may identify every shape in a diagram yet count one twice, overlook another, or lose track of which objects it has already examined. The authors of this paper argue that these errors are not always perception failures. They may reflect a separate problem: a reference gap between the model’s language-based reasoning and the visual entities, regions, or locations that reasoning concerns.
The central idea is straightforward. Reasoning is easier when objects have stable addresses. A numbered shape can be revisited as “shape 7” instead of “the one near the upper left.” A shaded angle makes its intended interior region explicit. Grid lines provide a coordinate-like frame for comparing positions, sizes, and transformations. The paper asks whether these external visual references can improve reasoning without retraining the model.
A controlled way to isolate reference
To study the issue, the authors introduce ReGap-Bench, a paired benchmark covering shape counting, angle ranking, size ranking, clock ranking, and transformation prediction. Each underlying scene is rendered in several versions. The objects, geometry, question, and answer remain unchanged; only the visual reference system varies. The benchmark adds numbering, shading, or grid lines without moving, removing, or covering the original content.
This paired design is important because it separates the effect of a reference from the difficulty of the image itself. If a model answers differently when only labels or grid lines have been added, the change offers evidence that the original failure involved reference management. The authors also test mismatched references, incomplete references, and models at different parameter scales.
The scaffolds are linked to visual routines: repeated operations such as indexing objects, keeping a region active, shifting attention, or tracing a boundary. Numbering is expected to help with indexing and marking. Shading should clarify bounded regions, such as the interior of an angle. Grids should support position, measurement, and tracking through transformations.
Task-matched scaffolds make the difference
Across the five tasks, the reported results support the authors’ main hypothesis: a reference helps most when it matches the visual routine required by the task.
- Numbering: On shape counting, gains ranged from 6.0 to 17.5 percentage points across the evaluated models. On clock ranking, gains ranged from 8.5 to 25.0 points.
- Shading: Shading improved size ranking by 4.0 to 11.0 points and improved angle ranking more modestly in exact-match accuracy. Pairwise ranking accuracy improved for four of five models on the angle task.
- Grid lines: Grids improved size ranking by 2.0 to 8.0 points and transformation prediction by 2.0 to 5.5 points.
The paper provides intermediate analyses that make these gains more interpretable. In clock ranking, numbering improved the rate at which several models correctly read individual clock times. For GPT-5.4 mini, the reported benefit mainly came from reducing the number of clocks for which the model stated no time, rather than substantially improving the accuracy of times it did state. In angle ranking, shading reduced “wrong-side” readings for several models, helping them attend to the intended region between the rays.
The authors also show that adding marks alone is not enough. On shape-counting scenes, numbering improved accuracy by an average of 11.3 percentage points, while shading produced an average gain of 1.7 points and grid lines reduced accuracy by 4.1 points. This pattern is consistent with task alignment rather than a generic advantage from visual clutter or extra information.
The gap persists in larger models
Model scale improved baseline performance, but it did not remove the need for references. Across six Qwen3-VL Thinking models ranging from 2B to 235B parameters, numbering continued to improve shape-counting accuracy. The reported gain varied from 9.0 to 23.5 points rather than shrinking consistently as models became larger.
A preliminary analysis further supports the distinction between perception and reference. On plain shape-counting images, Claude-Opus-4.7 reportedly counted all shapes, regardless of class, correctly in 96.5% of scenes and recognized individual cropped shapes correctly in 95.9% of cases. Yet it counted the requested shape class correctly in 72.0% of scenes. These results do not prove that perception was never involved, but they are consistent with a failure to maintain object references during counting.
A training-free agent skill
The authors turn the finding into a proposed inference-time skill. An agent identifies the visual operation required by a task, selects numbering, shading, grid lines, or no overlay, and draws the chosen scaffold on a copy of the image. In a 100-task sample from CartoMapQA, the skill selected an overlay for 57 tasks. The paper reports an overall improvement when the skill was available, including a 33.3-point gain on the small subset of shading tasks, 6.4 points on numbering tasks, and 4.2 points on grid-line tasks. Tasks requiring no overlay were unchanged in the reported table.
This is a practical design pattern rather than evidence that every visual problem should be annotated. A useful agent might first ask what must be tracked: discrete items, bounded regions, coordinates, distances, or transformations. It can then add the least intrusive scaffold that supports that operation.
What the evidence does not establish
The benchmark is controlled and synthetic, with 200 scenes per task, so its results may not transfer directly to photographs, documents, charts, or unfamiliar visual layouts. The models and prompting setup are also specific to the study. The CartoMapQA demonstration uses 100 sampled tasks, and the paper does not establish how the skill performs across a broader range of agents, overlay-generation tools, or real-world failure modes.
There are also evaluation caveats. Exact-match accuracy can hide partial progress on ranking tasks, while some response analyses rely on manual annotation or an LLM judge. The results show that references can help under the tested conditions; they do not demonstrate that the underlying model has acquired a general visual reasoning capability or that overlays are always safe. A mismatched scaffold can distract the model and reduce accuracy.
For practitioners, the paper’s strongest lesson is diagnostic: before buying a larger vision encoder or adding more chain-of-thought prompting, test whether the model can solve the same visual problem when its objects and regions receive stable, task-appropriate references. In some workflows, a small overlay may be a cheaper and more controllable intervention than retraining. But the reference must be designed around the task, validated on the target data, and treated as an aid to reasoning rather than a universal fix.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.