PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

PDRVLA: Enhancing Robot Manipulation with Physical Dynamics Representations

Robotic manipulation guided by vision and language instructions has seen significant advances through Vision–Language–Action (VLA) models. These models translate visual inputs and textual commands into robot actions, enabling robots to perform complex tasks. However, a critical challenge remains: understanding and anticipating the physical dynamics of objects during manipulation. Recognizing how objects move, change configuration, or interact physically is essential for precise and adaptive control.

The Problem: Limited Physical Dynamics Understanding in VLA Models

Current VLA models often struggle to encode physical properties and predict how scenes evolve as a robot interacts with objects. For example, changes in object mass or spatial relationships can affect manipulation success, but many models lack reliable adaptation to such variations. While pretrained video world models can capture scene dynamics by predicting future video frames, integrating them directly into control loops introduces latency, hindering real-time responsiveness.

Introducing PDRVLA: A Vision-Language-Action Model with Physical Dynamics

To address these limitations, the authors propose PDRVLA, a novel VLA model that incorporates a lightweight Physical Dynamics Module. This module extracts compact physical tokens from intermediate vision-language model (VLM) features, encoding information about future appearance, geometry, and motion without requiring online video generation during robot control.

The Physical Dynamics Module is trained with two complementary objectives:

  • Direct Prediction: It predicts future latent features representing RGB images, depth maps, and optical flow, encouraging the tokens to capture dynamic scene changes.
  • Auxiliary Supervision: A frozen pretrained video world model provides additional guidance by conditioning on the physical tokens to predict future video features, enhancing temporal representation learning.

During deployment, PDRVLA uses the learned physical tokens alongside final-layer VLM features to generate robot actions efficiently, avoiding the computational overhead of video generation.

Comprehensive Training Data and Annotation

The authors assemble a diverse training corpus entirely from publicly available datasets, integrating:

  • Real-robot trajectories capturing real-world dynamics and contact-rich interactions across various robot types.
  • Universal Manipulation Interface (UMI) data collected via handheld grippers and wrist-mounted cameras, offering embodiment-agnostic demonstrations.
  • Simulated interactions from multiple benchmarks covering standardized skills, long-horizon tasks, and bimanual or dexterous manipulation.
  • General and robotics-oriented vision-language (VL) data to enhance spatial, procedural, and task-level reasoning.

To further enrich the data, the authors apply:

  • Egocentric Human-to-Robot Synthesis: Adapting human egocentric demonstrations into robot-compatible trajectories and visual inputs.
  • Subtask Annotation: Segmenting long-horizon tasks into ordered subtasks with labels, enabling the model to understand task stages and transitions.
  • Embodied Chain-of-Thought (ECoT) Annotation: Adding detailed keyframe-level descriptions of scene context, task progress, and next motions to support reasoning and action planning.

Model Architecture and Action Generation

PDRVLA builds upon a Qwen3.5-4B VLM backbone. The Physical Dynamics Module uses 50 learnable queries to extract physical tokens from intermediate VLM layers, capturing dynamic information. These tokens, combined with final-layer VLM features and robot proprioceptive states, condition a Diffusion Transformer-based action expert that generates chunks of robot commands.

The model supports heterogeneous robot embodiments by standardizing action representations, including arm joint positions, end-effector poses, dexterous-hand joint positions, and gripper openings. Actions are predicted as relative changes over short chunks, enabling efficient replanning.

Training Methodology

Training proceeds in two stages:

  1. Physical Dynamics Pretraining: The Physical Dynamics Module is trained to predict future RGB, depth, and optical-flow latent features using direct supervision and auxiliary flow-matching losses from a frozen video world model. This stage encourages the module to encode rich physical dynamics.
  2. Joint Policy Training: The full model, including the VLM, Physical Dynamics Module, and action expert, is trained on vision-language-action and vision-language data. The direct future prediction losses are retained to maintain physical representation quality, while action and language objectives guide policy learning.

Evaluation and Results

The authors evaluate PDRVLA on four challenging simulation benchmarks:

  • LIBERO: Single-arm manipulation across spatial, object, goal, and long-horizon tasks.
  • LIBERO-Plus: Robustness evaluation of LIBERO tasks under various perturbations such as camera viewpoints, lighting, and noise.
  • RoboTwin 2.0: Bimanual manipulation tasks with clean and randomized settings.
  • SimplerEnv: WidowX robot manipulation on visual-matching tasks.

Across these benchmarks, PDRVLA achieves state-of-the-art success rates, including 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin 2.0, and 81.5% on SimplerEnv. These results demonstrate strong manipulation capabilities and robustness to environmental and instruction variations.

Component Analyses and Ablations

The authors conduct detailed studies to understand the contributions of various components:

  • World-Model-Assisted Pretraining: Incorporating supervision from the frozen video world model improves future feature prediction accuracy, particularly for RGB and depth modalities, compared to direct prediction alone.
  • Subtask Conditioning: Using self-generated subtask descriptions to condition action generation modestly improves success rates, especially under robot initial state and camera viewpoint perturbations. Updating subtask predictions every 1 to 8 action chunks balances performance and computational cost.
  • ECoT Supervision: Training with embodied chain-of-thought annotations enhances task success, but generating ECoT descriptions at inference does not consistently improve performance.
  • Auxiliary Object-Pose Supervision: Adding future object-pose prediction as an auxiliary task yields small gains in overall success, particularly under viewpoint perturbations, though some categories see minor decreases.

Limitations and Future Directions

While PDRVLA advances physical dynamics understanding in VLA models, the authors note several areas for future work:

  • Exploring predictive targets beyond RGB, depth, and optical flow, such as object-centric geometry, contact states, and material properties, to enrich physical representations.
  • Investigating how these additional cues complement each other and contribute to more effective action learning.
  • Maintaining a lightweight policy suitable for real-time deployment without incurring latency from video generation.

Practical Implications

PDRVLA’s approach of learning compact physical tokens that encode future scene dynamics enables more accurate and robust robot manipulation across diverse tasks and embodiments. By leveraging publicly available data and synthesizing human demonstrations, the model can generalize to varied environments and perturbations. This work paves the way for more adaptive robotic systems capable of anticipating physical changes during interaction, improving autonomy and reliability in real-world applications.

“The resulting policy uses physical tokens for action generation without running the video model during control,” the authors explain, highlighting the efficiency of their approach.

Overall, PDRVLA represents a significant step toward integrating physical dynamics understanding into vision-language-action frameworks, combining rich training data, innovative architecture, and rigorous evaluation to advance robot manipulation capabilities.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close