Your Benchmark Is an Empirical Measure Over Difficulty
Benchmark scores look objective because they reduce many model responses to a single number. But that number is also shaped by which questions the benchmark includes and how those questions are distributed across difficulty levels. In Your Benchmark Is an Empirical Measure Over Difficulty, Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, and Huan Zhang argue that this distribution should be treated as a first-class part of evaluation.
The authors’ central idea is simple: a benchmark is an empirical measure over difficulty. Every item contributes evaluation mass at its estimated difficulty, and the reported score is an average over that induced distribution. This framing exposes two ways a benchmark can give misleading evidence: measure mismatch and coverage gaps.
Why the distribution of questions matters
Measure mismatch occurs when a benchmark’s difficulty distribution differs from the distribution that an evaluator actually cares about. Imagine two models that perform similarly on easy questions but differ substantially on hard ones. A benchmark dominated by easy items may make the models appear tied, even if a harder target distribution would separate them clearly. The problem is not necessarily that the benchmark is “too easy”; it is that its weighting may be implicit rather than intentional.
Coverage gaps are a different failure mode. A model can improve substantially within a narrow range of difficulty while changing little elsewhere. If the benchmark contains few or no items in that range, its aggregate score may barely move. More repeated evaluations reduce response noise, but they cannot reveal improvement at difficulty levels where the benchmark has no observations.
Two diagnostics for benchmark quality
To formalize mismatch, Sun and coauthors represent item difficulties on a normalized interval from 0 to 1 and compare the benchmark’s empirical distribution with a target distribution using the 1-Wasserstein distance. They call this quantity Difficulty Distribution Mismatch, or DDM. A small distance means that the benchmark’s weighting is closer to the intended target; a large distance indicates that some regions receive more influence than others.
The paper’s theory shows that, when the difference in model performance across difficulty is sufficiently smooth, the Wasserstein distance bounds how far a benchmark-based model comparison can deviate from the comparison under the target distribution. If the target performance gap is larger than that bound, the model ordering is guaranteed to remain stable. The result does not make a high DDM automatically unacceptable: a non-uniform distribution can be appropriate when it reflects the intended application. It does mean that benchmark users should know what weighting they are accepting.
For coverage, the authors use the maximum gap between adjacent sorted item difficulties, including the endpoints of the difficulty interval. Their theoretical result identifies this maximum gap as an asymptotic minimax detection boundary for localized improvements. If an improvement interval is no wider than the largest uncovered gap, there is a worst-case improvement that no test can detect from that benchmark, regardless of how many repeated samples are collected. If the interval is wider than every gap, sufficiently many repeated evaluations can eventually detect it under the paper’s assumptions.
What the benchmark survey found
The authors estimate item difficulty with a one-parameter Item Response Theory model. Rather than treating an item’s average failure rate as its difficulty, IRT jointly estimates item difficulty and model ability from response data. Their study covers 25 benchmarks across math, coding, agentic tasks, logic, science, instruction following, and medicine.
The reported results suggest that both problems are common. Ten of the 25 benchmarks have DDM of at least 0.10 relative to a same-size uniform reference distribution. The authors emphasize that this is a diagnostic, not a universal quality threshold: it indicates strong implicit weighting when no other target has been specified.
Coverage gaps are also substantial. Eight benchmarks have a maximum gap of at least 0.10, and nine have a sum of their five largest gaps of at least 0.30. Small tests such as AIME24 and AIME25 show particularly large local gaps, but the issue is not limited to small benchmarks; the paper also reports notable gaps in benchmarks with hundreds of items. These results support a cautious interpretation of leaderboard differences and training curves, especially when progress is expected to be localized.
BENCHFILL: adding items where they are missing
The paper turns its diagnosis into a data-construction method called BENCHFILL. The algorithm targets the midpoint of a selected difficulty gap, chooses nearby existing items as rewriting seeds and references, and asks a language model to generate variants at the desired difficulty. Each candidate is then calibrated on the existing IRT scale, checked for validity, and the closest suitable item is retained for human review.
In experiments on eight benchmarks, the authors targeted the three largest gaps. With only three inserted items per benchmark, BENCHFILL reduced the sum of the three largest gaps by 21.09% to 38.88%, with an average reduction of 32.67%. The reported target deviation was consistently below 0.01. In a separate localized-improvement test on three AIME variants, the gap-filled version detected all three positive changes, while the original and baseline-augmented versions detected none. These findings demonstrate improved sensitivity in the tested settings, but they do not establish that every generated item will improve every benchmark.
Limitations and practical lessons
The difficulty scale is model-relative: it is inferred from selected model responses, not an intrinsic property of a question. Results can therefore change as model populations, prompting, or evaluation procedures change. The scalar 1PL IRT model also compresses potentially multidimensional abilities into one axis. The authors report that more flexible alternatives were unstable or less supported for their data, but that does not prove 1PL is appropriate for all benchmark families.
BENCHFILL also depends on reliable difficulty annotation, valid generated questions, and human review. Reducing a numerical gap does not by itself guarantee content diversity, construct validity, or resistance to contamination. The paper’s results are evidence for a useful auditing framework and a promising augmentation strategy, not a replacement for domain expertise.
For benchmark developers, the immediate recommendation is to publish difficulty-aware diagnostics alongside aggregate scores and to design item sets with deliberate coverage. For model researchers, leaderboard comparisons should be treated as comparisons under a particular empirical weighting, not as universal measures of capability. For training teams, a flat score may conceal localized gains; inspecting performance by estimated difficulty can reveal progress that a headline metric misses.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.