R^2Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

R²Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

Agent orchestration systems are increasingly expected to build workflows rather than follow a fixed sequence of prompts. A learned orchestrator may decide which agents to create, what roles they should play, how they hand work off, and which agent produces the final answer. The challenge is that training such systems usually depends on a static task collection and temporary skills that can hide whether the orchestrator has genuinely learned to solve a problem.

R²Orch, proposed by Mingda Zhang and colleagues, targets this problem with a pair of connected learning loops. The inner loop uses self-play between a Proposer that selects tasks and a Director that edits an agent graph. The outer loop manages the skill scaffold, testing whether individual skills are still useful and withdrawing them when the Director can perform without them. The authors describe the intended endpoint as zero-skill deployment: skills help during training, but the final Director should not need them at inference time.

Why ordinary training signals can fail

The paper identifies three weaknesses in existing approaches. First, a hand-built curriculum can restrict learning to tasks chosen in advance. Second, group-based reinforcement learning needs different outcomes among several rollouts. If every rollout succeeds or every rollout fails, the advantage signal can collapse. Third, a skill’s overall success rate mixes two effects: the difficulty of the selected task and the actual contribution of the skill. A high score therefore does not necessarily mean that the skill is still helping.

R²Orch addresses these issues by making the curriculum and skill evaluation interact. When a candidate skill is hidden from part of a rollout group, the Proposer is rewarded for selecting tasks whose skill-free rollouts still produce mixed outcomes. These tasks are neither already solved nor completely inaccessible, so they retain a useful training signal. The same randomized group supplies a paired comparison between runs with and without the skill.

Recursive Learnability and Skill Withdrawal

The paper’s first main mechanism, Recursive Learnability, measures outcome diversity after withholding the candidate skill. Its underlying score is proportional to 4p(1-p), where p is the success probability without that skill. The score is highest when successes and failures are balanced and approaches zero when all outcomes agree. The Proposer uses this reward to shift toward tasks that remain learnable as the scaffold shrinks, rather than tasks that are merely solvable with help.

The second mechanism, Skill Withdrawal, treats the hidden and visible rollouts as a controlled comparison. The authors estimate a skill’s effect by subtracting the success rate without it from the success rate with it. Evidence is accumulated in a Bayesian-bootstrap ledger. A skill becomes eligible for withdrawal only after a minimum number of records, and the paper’s decision rule considers the posterior expected harm of removing it. Withdrawn skills can also be retested and restored.

This design creates a feedback cycle: failures can be distilled into skills, skills can support training, and withdrawals expose new failures that guide the next tasks. The paper also uses a Director that edits an agent graph through validated atomic operations, including adding agents, setting prompts and models, defining layers, and specifying hand-offs.

What the experiments show

The authors evaluate R²Orch on twelve benchmarks spanning question answering, mathematics, code, and interactive tasks. Six are used for in-distribution training and testing, while six are held out for out-of-distribution evaluation. In the reported five-run results, R²Orch is the best-performing method on all twelve benchmarks, despite being tested with zero skills.

On the in-distribution suite, the method reports 91.09 exact match on HotpotQA, 86.09 on NQ-Open, 94.06 accuracy on MedQA, 81.33 on AIME 2026, 93.13 Pass@1 on MBPP+, and 90.63 success rate on ALFWorld. On the held-out suite, it reports 95.31 exact match on TriviaQA, 85.31 on MuSiQue, 86.56 accuracy on GPQA, 94.69 on MATH-Hard, 42.81 resolved issues on SWE-Bench, and 91.09 success rate on WebShop.

The ablations support the authors’ explanation of the system. Removing recursive self-play, removing the scaffold, using uniform task selection, replacing Recursive Learnability, or fixing the scaffold all reduce zero-skill performance. The authors report that all twelve withdrawals met their stated posterior criterion, with posterior expected positive harm no greater than 0.019 in the tested cases. They also report transfer to six other Worker backbones without retraining the Director, although the size of the gain varies by backbone and task.

Limitations and practical implications

The most important limitation is acknowledged by the authors: R²Orch begins with skills customized by hand for each dataset. The system can revise, withdraw, and distill skills, but automatically creating a strong initial skill library for a new domain remains unresolved. The results also come from benchmark tasks, simulated interactive environments, frozen Worker pools, and a particular training configuration. Success on these evaluations does not establish reliable self-improvement in open-ended real-world settings.

For practitioners, the paper offers a useful design pattern rather than a plug-in guarantee. If an agent system relies on procedural prompts or tools, evaluate them with matched with-and-without comparisons instead of a single aggregate success score. Keep training tasks near the boundary where rollouts disagree, because uniform outcomes provide little group-relative learning signal. Finally, make skill retirement reversible, versioned, and auditable; R²Orch’s ledger and retesting approach is intended to reduce the risk of silently removing a capability.

Overall, Zhang and colleagues present a coherent attempt to connect curriculum generation, skill evaluation, and zero-skill deployment. The demonstrated benchmark gains are substantial, but the broader claim of recursive self-improvement should be treated as conditional on better initialization, changing task distributions, and validation beyond curated benchmarks.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close