ParetoSaddler: Pareto-Guided Optimization of Agent Harnesses for Quality and Cost
Improving an AI agent does not always require changing the underlying language model. The surrounding harness—its prompts, tools, middleware, and control logic—can strongly influence whether an agent completes a task, how reliably it works, and how many tokens it consumes. The authors of ParetoSaddler examine how to optimize those harnesses when quality and execution cost matter at the same time.
The central problem is not simply choosing the best final configuration. It is deciding which intermediate candidates deserve more testing during the search. A candidate that is slightly more expensive today may fix a capability that enables later improvements. If an optimizer applies a strict deployment preference too early, it can discard that candidate before its longer-term value becomes visible.
Separating exploration from deployment
ParetoSaddler addresses this problem by separating two decisions. During optimization, it promotes candidates that show credible progress in either quality or cost. Only after the optimization budget is exhausted does it apply the deployment preference, such as a quality target, a cost constraint, or a request for several Pareto alternatives.
At each iteration, an underlying harness optimizer creates a candidate from execution traces and other evidence. The candidate is compared with its parent through repeated executions on screening tasks. Each run produces a quality result and a token-based monetary cost. ParetoSaddler then asks whether the candidate is supported as an improvement in at least one objective without an excessive regression in the other.
The method uses Bayesian bootstrap resampling rather than relying only on mean scores. It repeatedly reweights the observed executions, estimates the candidate’s quality and cost movement, and calculates how often the candidate satisfies the local promotion rule. A candidate is advanced to broader development evaluation only when that support reaches a chosen threshold. In the reported experiments, the threshold was 0.7 and the default evaluation used three executions per task.
Promoted candidates are retained in a development pool, while the optimizer maintains an empirical Pareto archive of non-dominated alternatives. This means the search can preserve both quality-oriented and cost-oriented directions instead of forcing every update toward one fixed trade-off.
Results on two agent benchmarks
The authors evaluated the approach on ClawMark and GAIA2 using domain-disjoint training, development, and test splits. The underlying model remained fixed while the harness was optimized. The quality-oriented Pareto solution improved the initial test result on ClawMark from 70.86% to 81.35%, while average cost fell from $1.17 to $1.01 per task. On GAIA2, quality rose from 56.22% to 62.89%, and cost decreased from $0.78 to $0.45.
The paper also compares ParetoSaddler with quality-only AutoSaddler optimization. The selected quality-oriented Pareto solution was 7.49 percentage points better on ClawMark and 5.56 points better on GAIA2, while also costing less in the reported tests. These findings suggest that cost is not merely a constraint that sacrifices capability. Cost analysis can expose unnecessary retrieval, delegation, polling, or tool-use behavior that also interferes with task completion.
Under quality-constrained cost minimization, ParetoSaddler selected solutions costing $0.82 per ClawMark task and $0.44 per GAIA2 task. The authors report that these results dominated the corresponding Meta-Harness variants in both quality and cost. The GAIA2 results were also reproduced in a second optimization run, with both runs improving on the default agent across the held-out universes.
Why the intermediate gate matters
The ablations provide evidence for the method’s main design choice. In one ClawMark example, a candidate reduced local cost but slightly reduced quality, so a quality-only gate would have rejected it. Broader evaluation later showed a better quality-cost point than its parent. In another trajectory, a candidate improved quality while increasing cost by 10.5%. A cost-focused gate would have discarded it, but Pareto-aware promotion preserved it for later refinement.
Repeated execution also mattered. In a ClawMark comparison, the three-run optimization reached stronger held-out operating points than the single-run version. The authors report that repeated runs exposed persistent failures, unstable outcomes, and expensive successful paths that a single execution could hide. Bayesian screening further improved the reported outcomes over a mean-based gate in a controlled ablation, filtering candidates whose apparent gains were driven by uneven or weakly supported observations.
Practical implications
For teams building long-running agents, the paper offers a useful operational pattern: treat harness optimization as multi-objective search, not just prompt polishing. Keep quality and cost measurements together, repeat evaluations when execution is stochastic, and avoid making the final deployment policy a hard filter during exploration.
In practice, this could mean retaining several harness versions for different operating points: a quality-first configuration for difficult cases, a lower-cost version for routine work, and a constrained option for workloads with a firm budget. The approach is especially relevant where agents use tools, external services, or multiple model calls, because wasted interaction and premature termination can affect both cost and task success.
Important limitations
The evidence is promising but bounded. The experiments cover two benchmarks and optimize specific harnesses and model configurations; they do not establish performance across all agent architectures or production workloads. Three executions per task leave rare behaviors potentially unobserved, and the Bayesian bootstrap addresses variation across observed runs rather than uncertainty from task sampling. Repeated evaluation also increases optimization expense. On GAIA2, the paper reports higher optimizer-side cost per patch for ParetoSaddler than for the quality-only and Meta-Harness comparisons.
The authors also explicitly state that they did not evaluate fairness, harmlessness, privacy-preserving mechanisms, or broader societal effects. Better benchmark quality and lower token cost should therefore not be interpreted as evidence of safety or unrestricted deployment readiness. ParetoSaddler is best understood as a search and evaluation strategy: a way to preserve promising paths while optimizing agent harnesses under competing objectives.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.