WorldBridge: Bridging Video and Action for Task Generalization in World Action Models

WorldBridge: Bridging Video and Action for Task Generalization in World Action Models

Robot learning has made impressive progress on familiar manipulation tasks, but a harder test remains: can a robot execute a task that was not represented in its action-labeled training data? The authors of WorldBridge study this question through World Action Models (WAMs), which combine visual prediction with action generation.

The central idea is that task generalization has two related but distinct requirements. A model must first understand how a new task should unfold visually. It must then translate that predicted visual evolution into executable motor commands. The authors call these capabilities video generalization and action generalization. The connection between them is a video-to-action bridge, closely related to an inverse-dynamics model.

Why video and action supervision play different roles

The paper’s clearest evidence comes from a controlled study on RoboTwin 2.0. With only in-distribution video-action supervision, the model achieved a 25.07% success rate on out-of-distribution tasks. Adding videos of those new tasks, but no corresponding action labels, raised out-of-distribution success to 55.90%. Adding just one action-labeled demonstration per new task increased it again to 70.70%.

These results suggest a practical division of labor. Additional video coverage teaches a model more about how tasks and scenes evolve, while action examples improve its ability to ground that knowledge in robot control. The authors report further gains as action supervision increases: five demonstrations per out-of-distribution task produced 84.67% success, and 20 produced 90.03%. At higher budgets, the returns diminished; 100 demonstrations reached 93.17%, compared with 92.83% for 550 demonstrations in the reported setting.

This is an important result, but it should be interpreted as evidence from a defined RoboTwin task partition, not as a guarantee that one demonstration will be sufficient for arbitrary real-world tasks. The value of sparse action supervision depends on whether the available video already captures the relevant visual dynamics and whether the learned bridge transfers to the new embodiment and environment.

How WorldBridge connects prediction to control

WorldBridge uses a Mixture-of-Transformers architecture with separate video and action experts. They retain modality-specific processing while exchanging information through joint attention. The video expert predicts future visual representations, and the action expert predicts robot action chunks. A structured attention design creates a directional pathway in which predicted future video can condition action generation.

The model is trained with joint flow matching. During inference, future video and actions are generated by progressively denoising their latent representations. The authors also introduce action forcing: during training, different actions within a chunk can receive different noise levels. This encourages the action expert to use surrounding temporal and visual context rather than treating each action as an isolated prediction.

Temporal information is handled at two scales. Short-term memory captures recent motion, while long-term memory represents broader task progress. This design is especially relevant to tasks where the current image does not reveal the complete state, such as remembering which objects have already been processed or preserving an execution order after the scene changes.

A data pipeline for heterogeneous robots and human video

WorldBridge is trained on robot trajectories and egocentric human demonstrations. The authors standardize these sources into a shared 46-dimensional action representation covering arm joints, end-effector poses, grippers, dexterous hands, and waist controls. Dimension-wise validity masks allow the system to ignore unavailable controls instead of treating missing values as meaningful actions.

The processing pipeline includes semantic auditing, coordinate and gripper normalization, trajectory trimming, kinematic checks, anomaly detection, and state-action consistency filtering. For egocentric data, hand keypoints are transformed into a common reference frame and converted into hand-derived action targets. This approach gives the model broader visual experience while preserving partial supervision across different embodiments.

Reported benchmark and real-robot results

Across the compared methods, the authors report strong results on several manipulation benchmarks. WorldBridge achieved a 96.23% average success rate on RoboTwin 2.0 in the full-data setting and 80.9% in the Clean-to-Random setting, where training uses clean demonstrations and evaluation includes randomized scenes. It reached 82.8% on RoboCasa GR1 TableTop, 83.0% on RMBench, 57.6% on DOMINO Level-1, and 84.4% on SimplerEnv-WidowX.

The ablations support the proposed design, although they do not isolate every architectural choice independently. Adding action forcing and the two memory scales improved the reported success rates across RoboTwin, RoboCasa, and RoboDojo. Removing both robot and egocentric pretraining reduced RoboTwin Clean-to-Random average success to 62.7%, compared with 80.9% when both sources were used.

On six tasks across AgiBot G2 and ARX AC-1 robots, WorldBridge achieved a reported average success rate of 72.5%. The tasks included flower insertion, drink scanning, cup hanging, LEGO separation, book packing, and screw sorting. The authors report that the model achieved the highest success on five tasks and tied the comparison model on the sixth. Screw sorting remained the weakest task, with a 55% success rate.

The system was also optimized for deployment. In the paper’s latency ablation, the reported median model-call latency fell from 957.2 milliseconds for joint video and action denoising to 103.9 milliseconds after decoupling, compilation, and CUDA Graph replay. The authors say the deployed system sustained nearly 7 Hz during real-world execution, including data processing and communication.

What the results mean for robot learning

The practical implication is a data strategy rather than simply a larger action dataset. For a new task, collecting broad video may be substantially cheaper or easier than collecting fully labeled robot trajectories. A small number of paired demonstrations can then help establish the action grounding needed to exploit that visual coverage.

Still, the evidence has boundaries. Most results are benchmark-based, and the real-world study covers six tasks and 20 trials per task. The paper’s comparisons depend on the selected baselines, training protocols, and task distributions. The authors also acknowledge that extending the video-to-action bridge to more diverse tasks with less action supervision remains open. Accordingly, WorldBridge demonstrates a promising route toward general-purpose manipulation, but not yet a solved recipe for reliable open-ended robotics.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close