Why benchmark compression is harder than it looks
Evaluating large language models, vision-language models, and agents on full benchmarks can be expensive. Many test suites contain hundreds or thousands of items, and agent or vision-language evaluations may require costly interactive rollouts. Benchmark compression addresses this problem by evaluating a smaller subset and using its scores as an approximation of full-benchmark performance.
The challenge is reliability. A random subset will usually be representative, but occasionally it can be unusually easy or difficult. Those rare failures matter when a compressed benchmark is used to compare models, make release decisions, or guide development. In “Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand,” George Ling and coauthors analyze this behavior and introduce BestRand, a lightweight method intended to reduce the risk of bad subset choices.
The paper’s explanation of random sampling
The authors model benchmark results as a matrix of item scores and study the error produced when a subset estimates the average score over the complete benchmark. Their analysis focuses on mean absolute error, or MAE, across models.
The central observation is that the error distribution is not uniform. Most random subsets produce relatively small errors, while a smaller number produce much larger errors. This creates a right-skewed, long-tailed distribution: random sampling is often good enough, but it is not consistently good enough.
The authors identify the common shift as an important source of this instability. A sampled subset may be collectively easier or harder than the full benchmark, causing many models to move in the same direction. Because modern foundation models often behave similarly on the same items, adding more models does not necessarily eliminate this shared error. In the paper’s theoretical model, positive behavioral correlation leaves a nonzero common-shift component even as the model panel grows.
This helps explain the title’s “almost.” Random sampling tends to concentrate around low error, but its occasional high-error outcomes remain a practical concern.
How BestRand works
BestRand keeps the simplicity of random sampling but adds a probe-guided selection step. The method first draws a candidate pool and runs a small number of probe models on that pool. It then creates multiple candidate subsets from the cached pool scores. For each candidate, BestRand measures how closely the average probe-score vector matches the pool’s reference vector, selecting the closest match.
The target models are evaluated only on the chosen subset. Candidate comparisons reuse cached probe results, so generating more candidates does not require additional target-model evaluations. In the authors’ main experiments, the default configuration used five probe models, 128 candidate subsets, and a pool four times the target subset size.
The theory motivates this design by decomposing the common shift into a component that probe scores can explain and a residual component that they cannot. Matching probe behavior should reduce the explainable part. The authors emphasize, however, that the strongest variance-reduction statements rely on approximations and an idealized selection model; they are not unconditional guarantees for every finite benchmark and candidate pool.
Evidence across text, multimodal, and agent benchmarks
The authors evaluate BestRand on more than 70 benchmarks: 44 text benchmarks, 16 multimodal benchmarks, and 10 agent benchmarks. They test retention ratios from 2% to 50% and compare BestRand with random sampling and several representative compression methods.
Across the main benchmark suite, the paper reports that BestRand reduces MAE by 16.4% on average relative to Random, ZipBench, and TinyBenchmarks, while reducing high-error risk by 50.2% against those three baselines. The reported construction cost is 51.3% of the average cost of the compared data-dependent methods. These are aggregate results across the authors’ chosen benchmarks, splits, retention ratios, and evaluation protocol—not a guarantee that BestRand will win on every individual task.
The method also performed well under the paper’s out-of-distribution tests. Across score-range, model-family, and temporal shifts, the authors report an average OOD error reduction of 21.8% relative to competitive baselines. Their extension to VLA evaluation on Bench2Drive and Fail2Drive likewise improved aggregate prediction error over random sampling in the reported settings.
Importantly, BestRand’s advantage is mainly about accuracy and stability, not ranking alone. Spearman rank correlations were already high for several leading methods, so the authors use MAE as their primary metric. Their plots and ablations show that BestRand generally lowers both average error and variation across random seeds, while gains from increasing the pool size or candidate count eventually diminish.
Limitations and practical implications
BestRand does not make compressed evaluation exact. The authors explicitly recommend full-benchmark evaluation for final assessments when it is feasible. A compressed subset is better suited to rapid iteration, exploratory comparisons, and situations where evaluating every item is prohibitively expensive.
The method also has a nonzero construction cost: probe models must score the candidate pool. That cost is small compared with evaluating every target model on the complete benchmark, but it may still be significant for expensive multimodal or interactive systems. Results can also depend on probe quality, pool size, subset budget, benchmark structure, and the similarity between construction and deployment conditions.
For practitioners, the paper suggests a pragmatic workflow. Use full evaluation for high-stakes conclusions, but use BestRand—or a carefully validated alternative—for frequent development checks. Measure not only average error but also seed-to-seed variation and high-error quantiles. Calibrate the retention ratio for each benchmark, validate the selected subset on held-out models, and avoid treating a low-cost compressed score as a replacement for uncertainty analysis.
The broader lesson from Ling and coauthors is that benchmark efficiency may not require elaborate learned selection systems. Random sampling captures much of the available signal; the more useful intervention may be a cheap mechanism that prevents its worst draws.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.