Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling
Mathematical modeling is a difficult target for AI evaluation because the task is not simply to solve an equation. A model must interpret an incomplete real-world problem, choose assumptions, build and run a computational model, inspect data and figures, and defend a numerical recommendation. The final answer may be plausible while violating a physical constraint, using the wrong units, or depending on an unreproducible modeling choice.
In this paper, Xiaojun Wu and coauthors introduce TB-MathModel, a benchmark intended to measure that complete workflow. It packages 18 recent CUMCM and MCM contest problems from 2023 through 2025 into terminal-based tasks. Each task includes the original statement, official attachments, and selected figures inside a container. An agent works in that environment and must produce a structured JSON artifact rather than only a polished report.
Why report grading is not enough
The authors argue that existing evaluations leave a crucial gap. Some benchmarks test mathematical reasoning or optimization fragments; others ask an LLM judge to grade a complete report. Report-level judging can reward organization and fluent explanation without verifying whether the reported numbers are correct or physically admissible.
TB-MathModel focuses on the numeric core of the final question. Its scoring compares an agent’s submitted values with reproductions of Outstanding-award papers, while also checking whether the result satisfies explicit invariants. The benchmark therefore separates two questions: how close is the answer to accepted human solutions, and is the answer internally feasible?
One-sided, multi-reference scoring
A single reference answer is a poor definition of truth in open-ended modeling. Different teams can make different assumptions and still produce defensible work. To address this, the authors assembled panels of Outstanding-award papers where available. The benchmark covers 99 papers in total, with 94 contributing scored quantities, and uses each paper as a separate reference when calculating the multi-reference O-EVAL score.
The scoring is also direction-aware. Depending on the metric, higher or lower values may be better, or the answer may need to approach a target. A model that meets or improves on an anchor should not be penalized merely because it differs from the reference. The authors also describe explicit rules for degenerate scoring bands, incompatible units, surplus items, and out-of-domain values. These details matter: their audit found that apparently minor scoring choices could systematically move results.
Feasibility is an independent gate
The benchmark’s most distinctive feature is its two-layer verification system. A programmatic gate covers all 18 tasks using six types of decidable invariants, including domain bounds, ordering relationships, exact identities, finite decision domains, and partition sums. An infeasible artifact receives a hard zero for the task score.
The second layer re-expresses the same invariants as 624 Lean 4 obligations. The authors report that these obligations are kernel-checked and that their trust audit found no use of sorryAx or native_decide. This does not mean Lean proves that a model captures the real world. Rather, it checks the narrower claims that a structured artifact can support, such as a probability remaining in range, a count being non-negative, or a stated identity holding.
That boundary is important. The authors distinguish unit and range validity, internal consistency, and physical truth. The first two can often be checked automatically. Whether a proposed design would actually produce a claimed amount of power, revenue, or performance may require an independent replay or evidence beyond the JSON artifact.
What the experiments found
The paper evaluates seven multimodal model-and-harness entries, using three seeds per task. The reported leader reaches 64.58% O-EVAL, followed by another flagship at 63.11%. A low-cost DeepSeek Flash entry scores 59.81% at a reported cost of $2.54 for an 18-task pass, compared with roughly $246 to $273 for the two leading entries. The authors present this as a substantial cost-performance tradeoff, not as evidence that the cheaper system is equivalent.
The results also show why rankings should be read cautiously. Per-seed variation reaches 2.93 percentage points, so nearby entries are not necessarily meaningfully separated. Changing the harness while holding the model fixed moved O-EVAL by only 0.07 percentage points in the reported comparison, suggesting that the benchmark was measuring model differences more than wrapper differences in that experiment.
Performance was uneven across problem families. CUMCM tasks, often more engineering-oriented and numerically constrained, were generally easier than the more open-ended MCM tasks. The authors report that additional reasoning compute was not automatically beneficial: one higher-budget DeepSeek configuration scored substantially below its default configuration.
The benchmark also caught its own mistake
The strongest case for verification comes from the dragon-dance task. The authors discovered that the reproduced paper anchor was physically infeasible: its reported speed and amplification ratio violated a 2 m/s identity. Several agents had produced the correct value but were initially penalized for failing to match the faulty anchor. After auditing the constraint and correcting the anchor, the benchmark’s judgment changed.
This episode demonstrates both the value and the limits of reference-based evaluation. A machine-checkable gate can expose an erroneous reference, but it cannot remove the need for careful benchmark construction.
Limitations and practical implications
TB-MathModel is still a small, pure evaluation set with no training, validation, or held-out split. Reference-panel sizes vary by task, some human papers disagree, and the benchmark scores only selected numeric fields rather than the entire modeling report. The verification layers cover structured evidence, not every claim about real-world physical validity. Unit ambiguity and schema design can also dominate scores, as shown by tasks where equivalent quantities were reported on incompatible scales.
For researchers, the paper’s practical lesson is to evaluate agents on complete, reproducible workflows rather than prose alone. Containers, fixed inputs, structured artifacts, executable checks, and explicit provenance make failures easier to diagnose. For model developers, the benchmark offers potential training signals: agents can be rewarded first for satisfying feasibility constraints, then for improving agreement with a diverse panel of accepted solutions. The authors’ broader claim is therefore plausible but still prospective: verifiable modeling traces may become useful both for measuring long-horizon competence and for training agents to check their own assumptions before presenting confident numbers.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.