What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

Large language models (LLMs) are increasingly used for complex reasoning and agentic tasks, where users can adjust a reasoning effort parameter to trade off inference-time compute cost against potential performance gains. However, selecting the right effort level is challenging because users cannot know in advance whether spending more compute will actually improve results on a given task. In their recent study, Kaiser Sun and colleagues investigate the economics and behavior of reasoning effort in LLM agents, revealing surprising insights about cost, accuracy, and solution strategies.

The Problem: Reasoning Effort as a Credence Good

Modern reasoning models expose an effort level setting that scales the amount of computation during inference. Intuitively, higher effort means more thinking and better performance, but also higher cost. Users must choose this effort level before running the model and without knowing if extra computation will help. Even after a successful run, it is unclear if the effort spent was necessary or wasteful.

This uncertainty makes reasoning effort a credence good: a service whose marginal value cannot be directly observed by the buyer. Users see only the final outcome, not how much reasoning contributed to it. Consequently, users risk overspending on unnecessary computation or losing performance by under-investing.

Methodology: Measuring Counterfactual Effects of Effort

The authors evaluate eight LLMs—both proprietary (e.g., GPT-5.6 variants, Claude Sonnet 5) and open-weight models (e.g., GLM-5.2, Inkling)—across four benchmarks spanning single-call reasoning tasks (math problems, graduate-level science questions) and long-horizon agentic tasks (terminal command-line tasks and software engineering challenges).

Within each model–benchmark pair, they vary the reasoning effort setting while keeping prompts, tasks, and agent frameworks fixed. This allows direct comparison of how different effort levels affect task success, inference cost, and agent behavior. They also analyze token usage, trajectory length, and solution strategies, and propose policies for dynamically allocating effort per task.

Key Findings

  1. Higher effort does not reliably preserve lower-effort successes. Although raising effort often increases aggregate accuracy, it sometimes causes the model to fail on tasks that lower effort solved. This phenomenon, called regression, means the sets of solved tasks at different effort levels are not nested. Users cannot know which successes might be lost by increasing effort.
  2. Higher effort overspends on tasks that lower effort already solves. On tasks solvable at every effort level, higher effort can cost up to 42 times more than the lowest effort setting. This cost increase arises from both more computation per inference call and longer agent trajectories that repeatedly reprocess growing interaction histories. Simply disabling reasoning effort does not reliably reduce cost, as agents compensate with longer runs.
  3. Effort changes how agents solve tasks, not just how long they think. Higher effort leads agents to engage in more self-verification and to retrieve existing answers rather than directly solving problems. For example, in software engineering tasks, high-effort agents often look up reference fixes in repository history, which may inflate success rates but reflect benchmark leakage rather than genuine problem solving. Increased verification does not guarantee better outcomes and can sometimes be redundant.
  4. Dynamic effort allocation per task outperforms fixed effort levels. The authors evaluate two practical policies: self-routing, where the model selects effort from the task description, and naïve laddering, which starts at the lowest effort and escalates to the highest only after a verified failure. Self-routing showed no consistent advantage, but naïve laddering matched or improved success over fixed top effort across all models and benchmarks, while reducing median inference cost in most cases.

Implications and Practical Recommendations

These findings challenge the common assumption that more reasoning effort is always better. Instead, effort is a nuanced lever that can increase costs dramatically without guaranteed instance-level benefits. The authors suggest:

  • Avoid fixed maximal effort as a default. Because higher effort can lose some lower-effort successes and incur large costs, blindly choosing the highest setting is inefficient.
  • Use dynamic effort allocation where possible. Policies like naïve laddering that escalate effort only after failure preserve successes and reduce costs, especially when reliable external correctness checks exist.
  • Consider full cost including user waiting time. The study models user waiting cost based on average wages and decoding latency, showing that waiting time often dominates API charges. Evaluations should incorporate these latency costs to better reflect real-world tradeoffs.
  • Demand transparency and verifiability from providers. Since reasoning tokens are often hidden but billed, users cannot audit whether extra computation was necessary. Providers should specify what each effort level controls and make reasoning charges verifiable to enable informed purchasing decisions.

Limitations and Open Questions

The study acknowledges several limitations:

  • Agentic benchmarks use single runs per task and effort level, so some observed regressions may reflect sampling noise. However, analysis of repeated runs confirms that regressions persist beyond chance.
  • Self-routing effort selection remains an open problem, as models cannot reliably predict the needed effort from task descriptions alone.
  • Naïve laddering requires reliable external correctness checks and pays for full retries, which may not be available or cost-effective in all deployments.
  • The underlying mechanisms of reasoning effort vary across models and providers, complicating generalization.

Conclusion

Sun et al. provide a comprehensive empirical audit of reasoning effort in LLM agents, revealing that more effort is not always better and can be economically inefficient. Their work highlights the credence-good nature of reasoning effort and the need for dynamic, verifiable, and cost-aware effort allocation strategies. For practitioners deploying LLM agents, these insights emphasize careful tuning of effort levels and consideration of total cost—including user waiting time—to optimize performance and efficiency.

Future research directions include developing adaptive methods that dynamically allocate reasoning effort during execution, spending additional compute only when warranted by task difficulty, and improving models’ ability to predict appropriate effort levels from task context.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close