Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

AI coding benchmarks often ask whether a model can generate a page, repair a bug, or implement a narrowly defined feature. Real product work is messier. Users ask agents to revise an existing application, interpret screenshots or media, preserve unrelated behavior, and propagate a change across components and state. Co-Design Bench is designed to measure that broader form of human–AI collaboration.

The problem: partial completion is not delivery

The paper’s central argument is that average feature coverage can hide an important failure mode. An agent may satisfy most requirements while leaving one critical interaction, route, data update, or preservation constraint broken. For a user, that is often an unfinished task, even if a benchmark awards substantial partial credit.

To study this gap, the authors assemble 273 tasks from real coding-session data. The source corpus contains 37,463 deduplicated queries drawn from 26,314 sessions. The final benchmark includes runnable applications before editing, normalized requests, available context, optional attachments, and hidden acceptance criteria written by experts.

The benchmark has two tracks. Multimodal Editing covers localized visual, structural, and content changes, sometimes grounded in images, audio, video, or documents. Repository Editing covers coordinated changes across frontend components, routes, state, data flow, and content. The latter is particularly important because a convincing screenshot does not prove that the underlying application works.

How the benchmark evaluates agents

Each task begins with a validated pre-edit application. The agent receives the request and relevant materials but not the hidden rubric. It can inspect the project, modify files, build the application, and interact with the result through a shared agentic harness.

The evaluation protocol is reference-free: it does not require gold code or a target screenshot. Instead, the evaluator compares the final application with the baseline and the requested outcomes. Criteria receive PASS, PARTIAL, or FAIL, scored as 1, 0.5, or 0 for graded completion. The authors also report strict completion, which counts only PASS, and Full-Pass Rate, which measures the share of tasks satisfying every criterion.

The paper emphasizes evidence matching. Source inspection can establish a static property, but it cannot prove that a feature is reachable through the rendered interface. Visual requirements require successful rendering and screenshots. Interaction requirements require recorded browser actions and resulting state changes.

After primary grading, a separate language-model verifier challenges PASS and PARTIAL decisions. It looks for stubs, hardcoded outputs, unreachable entry points, failures under varied inputs, and mismatches with the request. Evidence audits can invalidate evaluation when required renders or interaction logs are missing. Two frontend experts also reviewed rubric scores for 200 task–model outputs.

What the authors found

Across ten model endpoints, the strongest combined result came from gpt-6-astra, which achieved 69.9% mean strict completion. However, it fully passed only 26.4% of the 273 tasks. In other words, high criterion coverage coexisted with incomplete task delivery: most tasks still had at least one criterion that was not fully satisfied.

The distinction was sharper in Repository Editing. The leading endpoint reached 66.7% strict completion there but a Full-Pass Rate of 17.9%. Multimodal Editing was easier by these measures, with the same endpoint reaching 75.4% strict completion and a 41.0% Full-Pass Rate.

Complexity also mattered. Repository tasks involving three or four edit types—such as structural, stylistic, interactive, and content changes—had pooled strict completion of 59.1%, compared with 65.4% for tasks involving one or two types. The authors report that every tested endpoint scored lower on the broader group. This pattern is descriptive rather than causal, because the task groups differ in their requests and context.

Model rankings were not universal. The paper reports different relative strengths between the two tracks, and harness changes also altered results. In a separate comparison involving two solvers and three execution harnesses, the gap between the solvers varied by as much as 10 percentage points on Multimodal Editing and 16.2 points on Repository Editing. This suggests that a benchmark score reflects the model, the tool scaffold, and their interaction.

Why this matters in practice

For teams evaluating coding agents, the practical lesson is straightforward: do not rely on a single aggregate feature score. Track criterion-level coverage, all-requirements success, and evidence for important interactions. An agent that produces a polished interface may still fail to update dependent values, recover from a state transition, preserve existing behavior, or make a new route reachable.

Co-Design Bench also points toward better internal evaluations. Start from a known-working baseline, express requirements as observable acceptance criteria, and test both visual output and executable behavior. For compound changes, group criteria by capability—grounded understanding, UI transformation, and functional implementation—so failures reveal whether the problem lies in interpreting evidence, changing the interface, or wiring the application together.

Limitations and open questions

The benchmark is valuable, but its results should not be treated as a universal leaderboard. The task pool is curated from selected coding sessions, filtered for recoverability and verifiability, and consolidated into bounded requests. That makes evaluation feasible but may not represent all production software work. The paper also notes that modality and edit-breadth comparisons are not controlled causal experiments.

LLM-based judging remains another source of uncertainty, even with refutation, audits, and human checks. The authors report evaluator sensitivity and harness interactions rather than claiming to eliminate judgment error. Finally, tasks are evaluated as fixed requests; the benchmark does not simulate a continuing user conversation in which requirements change after seeing the first revision.

Its strongest contribution is therefore methodological. By separating partial requirement satisfaction from complete delivery, and by tying verdicts to source and browser evidence, Co-Design Bench offers a more realistic question for agentic coding: not “How much of the requested change appeared?” but “Did the application actually become the version the user asked for?”


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close