Your Policy Is Secretly a Harness: The Decision Geometry of Agent Interfaces

Your Policy Is Secretly a Harness: The Decision Geometry of Agent Interfaces

Language model agents are increasingly deployed through complex interfaces known as harnesses. These harnesses include prompts, tools, and control logic that shape how a model perceives tasks and executes actions. Jian Mu, Ruibo Shen, and Yao Shu investigate the subtle but profound ways in which a policy learned by a language model is influenced by—and in fact encodes—the characteristics of its training harness. Their work reveals why changing the interface can drastically affect agent performance and how adaptation can recover or even improve outcomes.

The Problem: How Does a Harness Shape a Policy?

When a language model agent is trained and deployed, it interacts with the environment through a harness that defines the observation format, available actions, and execution semantics. Typically, researchers evaluate new harness designs by freezing a trained model and measuring its performance under the new interface. This approach assumes that the policy learned under one harness transfers seamlessly to another. However, the authors observe that this assumption often fails: the same model can perform well under one harness and poorly under another, even when the underlying optimal decisions remain unchanged.

This raises fundamental questions: What exactly does a policy learn from its harness? What aspects of that learning survive when the harness changes? And how can we predict or improve transfer performance?

The Method: Harnesses as Regularized Control Problems

The authors formalize a harness as an interface that encodes the environment state into a context, decodes model outputs into semantic actions, and executes those actions. This induces a semantic control problem where the policy chooses actions to maximize a regularized return, balancing task rewards against divergence from a reference policy.

Under this framework, the optimal policy has a softmax form over action scores that combine long-term value and the reference policy’s log probability. Crucially, the log odds of the optimal policy reveal the relative preferences that the harness imposes on actions. Thus, a policy trained to optimality under a harness implicitly encodes the harness’s preferences as these relative action scores.

The authors distinguish two key quantities:

  • Ceiling (C): The best achievable regularized return under a harness, independent of the model’s weights.
  • Mismatch (M): The return lost by the policy due to deviations from the optimal policy, measured as a KL divergence weighted by the policy’s occupancy.

Training changes only the mismatch by improving the policy, while changing the harness can alter both the ceiling and the mismatch.

Key Evidence and Results

1. Post-training Makes Preferences Harness-Specific

Using MiniWoB++ web tasks, the authors post-train a Qwen3.5-2B model under six different harnesses. They find that training moves the model’s action preferences closer to the optimum of the training harness and makes these preferences specific to that harness. When deployed under other harnesses, the model’s preferences deviate more from the respective optima, confirming harness specificity.

2. Presentation Changes Can Break Transfer Even When Optimal Decisions Are Unchanged

In a striking experiment, the authors reverse the order of observation fields in the input JSON without changing the underlying task, actions, or rewards. Although the optimal policy remains the same, the trained model’s success plummets from 85% to 19%. This failure is attributed to a shift in the model’s readout—the way it expresses its learned preferences through the new interface—rather than a change in the optimal decisions themselves.

3. Deployed Return Equals Ceiling Minus Mismatch

The authors prove that the regularized return a policy achieves under a harness equals the harness’s ceiling minus the policy’s mismatch. This identity explains why a new harness that raises the ceiling can initially hurt a frozen model by increasing mismatch. However, further training (adaptation) under the new harness reduces mismatch and can ultimately yield better performance than the original harness.

4. Adaptation Recovers Gains from Harmful Harness Revisions

They formalize a threshold condition: adaptation will improve performance beyond the original if the mismatch after adaptation falls below the new ceiling minus the original return. Experiments with batched clicks (a harness that executes two clicks per decision) illustrate this. Although the frozen model performs worse under the new harness, adaptation reduces mismatch sufficiently to surpass the original model’s performance.

5. Predicting Transfer Outcomes from Ceiling and Mismatch

By measuring ceiling and mismatch before adaptation, the authors can predict which harness changes will ultimately benefit the agent after adaptation. They validate this across multiple harness swaps and show that these metrics outperform naive frozen-model comparisons.

6. Estimating Mismatch from Sampled Actions

Since exact optimal policies and full action distributions are often unavailable, the authors develop methods to estimate mismatch from sampled actions and a teacher model as a stand-in for the optimum. Applied to coding-agent harnesses on Terminal-Bench Lite, these estimates correctly rank harnesses by accuracy, demonstrating practical utility.

Limitations and Considerations

  • Dependence on Reference Policies and Regularization: The framework relies on KL-regularized control with a fixed reference policy. Real-world scenarios may involve more complex or changing references.
  • Finite State and Action Spaces: The theoretical results assume finite spaces and known transition dynamics, which may not hold in all applications.
  • Teacher Model Approximation: Estimating ceilings and mismatches via a teacher model is an approximation; its quality depends on the teacher’s optimality.
  • Focus on Post-Training: The study centers on post-training adaptation rather than joint harness-weight optimization, which may behave differently.

Practical Implications for AI Agent Design

This work provides a principled lens on how interface design and training interact in language-model agents. Key takeaways include:

  • Policy Encodes Harness Preferences: A trained policy carries the relative action preferences of its training harness, making it sensitive to interface changes.
  • Frozen Model Comparisons Are Insufficient: Evaluating harnesses by freezing a model trained under a different harness confounds the harness’s potential with the model’s mismatch.
  • Ceiling and Mismatch Metrics Guide Harness Design: Reporting both the ceiling and mismatch alongside frozen returns helps disentangle harness quality from model adaptation.
  • Adaptation Unlocks Harness Gains: Harness revisions that initially degrade frozen-model performance can become beneficial after adaptation reduces mismatch.
  • Sample-Based Mismatch Estimation Enables Monitoring: Measuring mismatch from sampled actions allows practical tracking of harness and policy alignment during deployment and adaptation.

Overall, the authors’ framework and empirical findings suggest that harness design and model training should be co-optimized, with careful measurement of ceiling and mismatch to guide improvements. This insight is especially relevant as language-model agents become more complex and interface-rich.

In the words of the authors: “A policy trained under a harness learns the harness’s relative action scores and expresses them through its interface. Post-training makes these preferences harness-specific, and a new interface can change the model’s choices even when every optimal decision stays the same.”


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close