Why an agent’s harness matters
Large language model agents do not act from model weights alone. Their behavior also depends on the surrounding harness: prompts, tool interfaces, context and memory handling, and the rules that turn model outputs into actions. A harness can encourage an agent to inspect a schema, paginate through every result, verify a write, or reserve enough execution budget to submit a task.
Harness-S: Learning from Sibling Harnesses for Generalizable Agent Training asks whether these execution-level improvements can teach the policy itself, rather than remaining permanently dependent on a specialized runtime. Kangning Zhang and colleagues describe a system for using several related, purpose-built harnesses to rescue failures, then training one shared policy that is evaluated again with the original base harness.
The problem with searching for one perfect harness
The authors identify three properties of harness evolution. First is locality: an edit may repair one class of failure while causing regressions elsewhere. Second is complementarity: different sibling harnesses can solve different failures, so their combined coverage may exceed that of any individual variant. Third is nonstationarity: a harness that helps an early version of a policy may become less useful, or even harmful, as training changes the policy.
These properties make a universal harness difficult to construct. Combining multiple local fixes can create interference, while repeatedly evolving the harness alongside the policy can be expensive. It can also produce gains that disappear when the policy returns to the original execution environment. Harness-S instead preserves multiple specialized variants and uses them selectively during training.
How Harness-S works
Building a sibling pool
The first stage, called Directional Harness Evolution, starts from a common base harness and diagnoses failures in execution trajectories. Candidate edits target distinct groups of related root causes. Valid, executable variants are retained in a Sibling Harness Pool, while parent selection favors variants that collectively cover more validation failures.
The pool is fixed during policy training. This is an important design choice: the implementations are not continually changed as the policy evolves. The evidence used to decide which variant is useful can change, but the available harnesses remain a reusable collection of alternative repair strategies.
Rescuing failed tasks
During Complementary Harness Rescue Training, the policy first attempts a task several times under the base harness. Auxiliary harnesses are invoked only when all of those initial attempts fail. An Adaptive Harness Router selects the top-k sibling variants, and the same policy makes fresh attempts from the task’s initial state under those variants.
If the rescue attempts produce useful outcomes, they replace the all-failure group used for reinforcement learning. This gives the group-relative optimizer a contrast between successful and unsuccessful trajectories— a signal that is absent when every initial attempt fails. The authors use GRPO-style updates and optimize policy tokens while preserving the context in which each trajectory was generated.
Routing that adapts to the policy
The router uses Harness Behavior Profiles describing when a variant tends to help or hinder the current policy. Multiple language models independently rank candidates using these profiles, validation outcomes, and evidence from similar tasks. Because harness utility changes during learning, the profiles are periodically re-anchored with trajectories from the current policy.
The same router can also be reused at inference time. In that setting, Rank-1 routing chooses a harness for each task before execution, rather than using one fixed harness for every task.
What the experiments found
Zhang and colleagues evaluated Harness-S with Qwen3.5-4B and Qwen3.5-9B on AppWorld, SWE-Gym, and DBBench. The main comparisons included policy-only training, base-harness rescue, fixed harness mixtures, frozen-policy harness evolution, and methods that jointly evolve harnesses and policies.
The reported results favor Harness-S under both the original base harness and task-specific routed execution. For example, the 9B policy achieved 72.52 Pass@3 on AppWorld, 40.00 on SWE-Gym, and 76.33 on DBBench under the base harness. With Rank-1 routing, the corresponding figures were 75.19, 43.75, and 78.00. The paper reports these results as means across three randomized runs, with standard deviations for Harness-S.
The gains were not limited to adding more random attempts. On the AppWorld 4B training comparison, Harness-S produced effective groups containing both successes and failures more often than standard GRPO and base-harness rescue. The authors argue that targeted rescue supplies better learning signals than either repeated sampling under the same harness or a fixed mixture of harnesses.
Behavioral analyses also support, but do not by themselves prove, policy transfer. Under the base harness, the trained policy more often looked up schemas, paginated through sources, verified results, and recovered from blocked software tests. The paper describes held-out examples in which behaviors first elicited by sibling harnesses—such as applying an intersection constraint or completing a multi-step account-recovery procedure—reappeared under the base harness on new instances.
Cross-harness tests provide additional evidence. The Harness-S policy retained its advantage over the reported GRPO baseline under external harnesses without further policy updates, although it remained sensitive to which execution interface was used. Rank-1 routing delivered further gains at the same executor rollout count in the reported evaluations.
Limitations and open questions
The evidence is substantial but bounded. The experiments cover three benchmarks, two model sizes, and a particular family of harnesses descended from a shared base. Results may differ for other models, environments, verifiers, or independently developed execution systems.
Harness-S also has nontrivial resource requirements. The authors report additional rescue rollouts, GPU time, and external-model token use for profile construction, routing, and harness evolution. Although the method compares favorably with several baselines on selected costs, the pool still must be created and calibrated. The paper’s behavioral frequencies and case studies are evidence of changed execution patterns, not a complete causal account of which internal capabilities were learned.
Practical implications
For teams training long-horizon agents, the central lesson is to treat execution support as a source of diverse experience rather than only as a deployment configuration. A useful workflow may preserve specialized harnesses, invoke them only after the base system fails, and evaluate the resulting policy under the simpler runtime that will actually be deployed.
The approach is especially relevant when failures have recognizable causes—pagination mistakes, weak schema use, incomplete verification, or brittle recovery procedures. However, practitioners should measure rescue cost, router calibration, and performance under the target harness separately. Harness-S suggests that a collection of imperfect specialists can teach a more general policy, but the authors’ results do not establish that every harness improvement will transfer.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.