World In-Context Learning for Agentic Robot Manipulation

A different way to teach robots manipulation

Robot-learning systems usually acquire skills by changing their network weights. That approach can work well when training and deployment look alike, but it often struggles when lighting, textures, object arrangements, or physical conditions change. In-context learning offers the opposite trade-off: a frozen model can adapt from a demonstration without updating its weights, yet a demonstration from another scene gives little information about what will happen in the robot’s current world.

In World In-Context Learning for Agentic Robot Manipulation, Heng Fang, Zehui Guo, Yuehan Zhang, Mingyang Du, Dingkang Liang, and Xiang Bai propose a bridge between these ideas. Their central claim is that a robot can learn more effectively in context if it is allowed to practice in a world—typically a simulator—and record the resulting feedback in a readable memory rather than in model parameters.

From in-context learning to world in-context learning

The authors call the paradigm World In-Context Learning, or WICL. The process begins with one demonstration. A frozen foundation model interprets it, attempts the task in a world, observes outcomes, and revises its context. Once rehearsal is complete, the resulting context is frozen and evaluated on previously unseen instances.

This distinction matters. Ordinary in-context imitation can describe what happened in the demonstration, but it cannot directly reveal why an action would fail in a new layout. WICL adds the missing feedback loop. The robot can discover, for example, that a grasp specification is too fragile, that a placement needs a relative offset, or that a recovery action is required after losing contact.

The authors argue that this learned information is primarily semantic: object roles, task stages, relative geometry, and lessons from failures. Because it is not tied to the exact pixels of a rehearsal scene, it should transfer across visual distributions and, in principle, across worlds with shared task semantics.

How RoboScript implements the idea

To test WICL, the authors build RoboScript, a modular infrastructure for agentic manipulation. It separates the system into an agent plane, which acts through tools, and a control plane, which manages protocols, evaluation, memory, and privileged simulator information.

The learned artifact is a structured Task Memory. It records object roles, workflow stages, grasp and placement specifications, parameters, and lessons from previous attempts. A memory linter rejects scene-specific coordinates, simulator ground-truth terms, and instance identifiers. This is intended to prevent the memory from quietly encoding information unavailable to a real robot.

Learning has two main stages. First, imitation creates an initial memory from the demonstration. Second, rehearsal tests that memory on new simulator scenes. The acting agent receives deployable observations such as RGB-D images, proprioception, and action receipts. A reviewing role can diagnose failures during learning and revise a cached memory, but revisions are accepted only when they improve performance on the relevant instance.

At deployment, the frozen memory supports scene anticipation, arm selection, completion checks, and recovery without resetting after every failed action. The authors also add a reflection loop: after evaluation blocks, a reflection agent reviews deployable records and writes guidance for later, disjoint blocks. This is presented as recursive self-improvement without gradient updates.

Results on RoboTwin 2.0

The main evaluation covers all 50 tasks in RoboTwin 2.0, with clean scenes and scenes containing randomized textures, lighting, table height, and distractors. RoboScript rehearses on clean simulator scenes, uses one demonstration per task, and performs no gradient updates.

With GPT-6.1-Sol as its foundation model, RoboScript reaches 83.4% mean success across the clean and randomized settings: 83.6% on clean scenes and 83.2% on randomized scenes. The paper reports that the strongest trained action policy in its comparison reaches 61.7% on the same combined measure despite using 50 demonstrations per task. The authors therefore attribute a gain of more than 20 percentage points to the combination of a frozen model, world rehearsal, and contextual memory.

The robustness result is especially notable. RoboScript retains between 95.2% and 100.8% of its clean success across the three tested foundation models, while the compared trained policies retain at most 65.1%. The authors’ failure analysis suggests that randomization does not substantially increase knowledge failures; remaining errors are more often related to perception or physical execution.

Performance also varies with the foundation model. GPT-5.6-Luna reaches 77.9% mean success, and DeepSeek-V4.1-Flash reaches 78.1%, both above the listed trained policies. Reflection produces net gains across the reported rounds, although those gains become smaller as easier failure modes are addressed.

Evidence beyond simulation

The authors report a limited cross-embodiment test on a physical dual-arm PiPER robot. Memories rehearsed in simulation on a different robot are adapted with a lightweight remapping of coordinate conventions, kinematics, and gripper parameters. Across four tasks and 10 episodes per task, success counts are 10 of 10, 7 of 10, 7 of 10, and 6 of 10. The paper attributes failures mainly to gripper alignment and depth-estimation errors.

This result supports the authors’ semantic-transfer hypothesis, but it is not evidence that the approach is solved for general physical manipulation. The test covers four tasks, uses offline action planning, and involves human supervision on the real robot.

Limitations and practical implications

RoboScript’s strongest advantage comes with substantial costs. The authors report that a GPT-6.1-Sol deployment episode averages 54.7 model calls, 2.33 million tokens, and 149 simulation steps. The reported cost for learning all 50 tasks and evaluating 1,000 episodes is 1,911 US dollars at the cited list API prices; the paper also notes that API quality can change over time and shift success rates.

Latency is another practical obstacle. A trained policy can execute rapidly, whereas an agent that waits for repeated foundation-model calls may be too slow for safety-critical or time-sensitive work. The current evaluation also focuses on rigid or articulated objects. The authors identify deformable objects, cables, and tightly coordinated contact as important unresolved challenges.

For practitioners, the paper suggests a useful design pattern: use simulation not merely to train policies, but as a test-and-feedback environment for a frozen agent. Store lessons in an auditable, semantic memory; enforce a strict boundary between privileged learning data and deployable observations; and evaluate on disjoint instances to separate genuine generalization from memorization. The authors’ results demonstrate that this pattern can be effective on a demanding benchmark. Whether it will remain reliable, affordable, and fast enough for broad real-world deployment remains an open question.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close