RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
Robotic manipulation and control face a fundamental challenge: how to learn and predict world dynamics across diverse embodiments, such as human hands, dexterous robot grippers, and simulated agents, each with incompatible action spaces. Traditional robot world models rely on native robot actions, which differ drastically between platforms, limiting scalability and generalization. Meanwhile, large-scale human interaction videos offer rich visual data but lack explicit action labels, complicating their use for robot learning.
The Problem: Cross-Embodiment Action Conditioning
The core difficulty addressed by the authors is how to build a world model that can learn from heterogeneous video data—human videos, robot demonstrations, and simulations—without requiring a common, aligned action space. Existing approaches either depend on robot-specific control signals or coarse task-level language instructions, which are insufficient for precise, frame-level control needed in contact-rich manipulation tasks.
Language-conditioned video models capture semantic goals but cannot specify exact trajectories or timing, making them unsuitable for fine-grained control. Directly conditioning on raw robot actions is problematic because different robots have different joint configurations, control frequencies, and coordinate conventions, creating an alignment bottleneck.
Method: Latent Actions as a Universal Interface
The authors propose RynnWorld-Latent, a video world model conditioned on universal latent actions—continuous, 3D-grounded descriptors extracted from visual transitions that abstract away embodiment-specific control details. These latent actions, derived from the RynnLAM encoder, describe what changed between consecutive video frames in a shared latent space.
This latent-action interface enables joint pretraining on a massive corpus of heterogeneous videos without requiring aligned native action spaces. The model predicts future video latents conditioned on an initial observation and a sequence of latent actions, effectively modeling visual dynamics across embodiments.
Technically, RynnWorld-Latent builds on a pretrained video diffusion transformer backbone (Cosmos3-Edge) and incorporates two conditioning pathways: global action tokens participate in self-attention, while a local, frame-aligned modulation pathway injects action information directly into corresponding future video latents. This design leverages the temporal alignment between latent actions and video latent frames, allowing precise conditioning on motion transitions.
Data Generation Pipelines
Beyond future prediction, RynnWorld-Latent serves as a data-generation engine for scalable robot learning through two complementary pipelines:
- Digital Teleoperation: After post-training adaptation to a target robot, the model can be conditioned on native robot actions to generate synthetic video demonstrations interactively. Operators provide control signals retargeted to the robot action space, and the model generates corresponding video rollouts, enabling data collection without physical robot execution.
- Latent-Action Data Synthesis: Given a source human video, the latent-action sequence is extracted and paired with an edited first frame that replaces the human embodiment with a target robot or environment. Conditioning on this edited frame and latent actions, the model synthesizes a new video rollout depicting robot-like interactions. Pseudo robot actions are obtained by retargeting and inverse kinematics, producing paired action-video data for downstream training.
Evidence and Results
The authors pretrain RynnWorld-Latent on an extensive corpus of 23 datasets totaling over 6,000 hours and 553 million frames, including human egocentric videos, robot demonstrations, and simulation rollouts.
Video Prediction Benchmarks: On public benchmarks such as DreamDojo and on two real-robot platforms (Marvin-WUJI with dexterous hands and Astribot-S1 mobile manipulator), RynnWorld-Latent achieves superior video prediction metrics (PSNR, SSIM, LPIPS) compared to prior state-of-the-art models. Ablation studies confirm the importance of latent-action conditioning and the local frame-aligned injection pathway.
Qualitative Action Following: The model generates distinct future video rollouts when conditioned on different latent-action sequences but the same initial observation, demonstrating controllable visual dynamics and cross-embodiment motion transfer.
Policy Learning Improvements: Synthetic data generated via both digital teleoperation and latent-action synthesis improves downstream robot policy success rates. On real-world tasks spanning precise plate placement, bimanual plate wiping, steak serving, and plate retrieval, policies trained with augmented synthetic data outperform those trained on real demonstrations alone. Similarly, synthetic mid-training on the large-scale latent-action corpus improves success rates on simulation benchmarks LIBERO and LIBERO-Plus.
Limitations and Considerations
The authors acknowledge several limitations:
- Latent actions encode visual transitions and may include camera motion or appearance changes unrelated to physical control, so qualitative action following and image-based metrics do not guarantee causal correctness or physical validity.
- The synthesis pipeline combines independently generated observations and retargeted pseudo-actions, which may not be physically consistent despite temporal alignment and filtering.
- Reported real-world success rates are based on limited trials (35 per task), so small improvements should be interpreted cautiously.
- Synthetic mid-training adds both data and compute, and the relative contribution of each is not isolated.
Practical Implications
RynnWorld-Latent introduces a scalable approach to learning and generating controllable visual dynamics across heterogeneous robot embodiments and human videos. By using universal latent actions as a shared interface, it enables leveraging vast unlabeled human video data alongside robot demonstrations and simulations, overcoming the alignment bottleneck of native robot actions.
This approach facilitates synthetic data generation for robot policy training without costly physical data collection, supports interactive digital teleoperation for data augmentation, and enables cross-embodiment motion transfer through first-frame editing. The demonstrated improvements in video prediction fidelity and downstream policy success suggest that latent-action-conditioned world models are a promising direction for generalist robot learning.
Future work will need to address long-horizon consistency, causal alignment between actions and observations, and broader transfer across diverse embodiments and environments.
“These findings support shared latent actions as a practical interface for cross-embodiment world-model learning and controllable robot-data generation.”
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.