SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

System One models are designed to make structured decisions in a single forward pass. Instead of producing a long chain of reasoning, they select among predefined outcomes, assign a score, or answer a binary question. That design can make them substantially faster than reasoning-oriented System Two models, which is attractive for routing, classification, tool selection, and other high-volume workflows.

But speed and average accuracy do not fully answer whether a decision model is safe to deploy. Will it make the same choice if answer options are reordered? Will it recognize that none of the available options is correct? And can it handle increasingly difficult tasks or decisions that depend on a changing environment?

In SO-Arena, Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, and Xinchao Wang introduce a benchmark intended to measure both capability breadth and decision reliability. The authors evaluate 21 models across 2,919 tasks, 11 domains, and five capability families.

A benchmark built around more than accuracy

SO-Arena draws from 40 public datasets, benchmarks, and environments. Its coverage includes semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion. The tasks include familiar static decisions, such as sentiment classification and mathematical multiple choice, as well as environment-based decisions involving web interaction and tool use.

The authors emphasize evidence and label quality. They retain traceable source information, manually review references alongside language-model review, and check that labels are supported by information visible to the evaluated model. Reference material is kept separate from model inputs. Some tasks also use verified execution or replay, including code-understanding checks and interactive tool episodes.

To expose differences between models that a single average might hide, nine domains receive domain-specific difficulty tiers. These tiers are relative to each domain rather than calibrated as a universal easy-to-hard scale. The benchmark also distinguishes two roles in agentic settings: selecting actions that advance a goal and verifying whether a recorded execution actually satisfied its requirements.

Reliability is measured separately

The paper’s central argument is that capability and reliability should be reported as complementary dimensions. SO-Arena therefore tests reliability in two controlled ways.

  • Option-order stability: the same item is shown in its original order and in 12 reorderings. The model should select the same option based on its content, not its position.
  • Missing-answer recognition: the correct option is removed from a copy of an item. The model should recognize that no offered answer is valid rather than confidently selecting a remaining distractor.

The combined reliability score uses the geometric mean of these two components and is positive only when both perform above chance. This design matters because consistency alone is not correctness. A model can repeatedly choose the same wrong answer, while a model that performs well on ordinary items may still fail to notice when the answer set is incomplete.

What the evaluation found

The authors report a clear capability-versus-latency trade-off. The two System Two reference models lead the overall capability leaderboard, while the strongest System One models trail them by roughly ten points in the reported comparison. At the same time, every open-weight System One model completed the timed tasks in under 0.3 seconds per task on one H100 GPU, according to the paper—tens of times faster than the two System Two references in the reported setup.

Scaling and additional reasoning did not help uniformly. Larger backbones produced their largest gains in grounded decisions such as safety, tool use, web interaction, and task verification. Explicit reasoning helped most on mathematics and scientific reasoning. In the authors’ analysis, an automatically thinking 4B model rose from 50.8 to 92.5 on the selected mathematics comparison, but thinking offered little benefit in several direct semantic tasks.

The most important result concerns reliability. The strongest-performing System One model was not the most reliable. JevAny-27B ranked highest on the paper’s reliability measure while ranking only eighth among the 18 System One models evaluated for reliability. The authors also find that option-order stability generally follows task performance, but missing-answer recognition varies much more widely among models with similar capability scores.

Why the findings matter for deployment

For practitioners, SO-Arena suggests that a fast decision model should not be selected from an accuracy leaderboard alone. A model used for ticket routing, policy screening, tool invocation, or workflow verification should be tested under the perturbations it will face in production.

  • Reorder choices and check whether the selected outcome changes.
  • Remove the valid option and measure whether the system can abstain or flag the case.
  • Separate direct classification from decisions requiring evidence, history, or multi-step state changes.
  • Evaluate hard cases by domain instead of relying only on an aggregate score.
  • Measure latency alongside quality, since a slower model may be preferable when errors carry higher operational costs.

These tests can also guide routing strategies. A System One model may be a strong fit for routine semantic judgments, while difficult mathematical, scientific, or long-horizon tasks may benefit from escalation to a reasoning model or a human reviewer. That is a practical implication of the paper’s results, not a universal rule established by the benchmark.

Limitations and open questions

SO-Arena is broad, but its conclusions remain tied to its construction. The benchmark combines existing datasets with derived tasks, so source-label quality, task transformations, and domain coverage influence the results. Difficulty tiers are domain-relative and should not be interpreted as directly comparable across domains. Some slices are small; notably, the hard tier for task-completion verification contains 20 items in the analysis reported by the authors.

Latency results also depend on model versions, serving paths, hardware, and API conditions. The reliability score measures two specific forms of robustness; it does not establish calibration, factuality, adversarial resistance, or real-world safety by itself. Finally, some references are model-derived, although the authors flag such metadata and describe review procedures.

Even with those qualifications, SO-Arena makes a useful case for evaluating decision models as operational components rather than merely as fast classifiers. Its central lesson is straightforward: capability tells us what a model can usually do, while reliability asks whether its decisions remain dependable when the choices, evidence, or task state change.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close