Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

Vision-language-action (VLA) policies promise a practical route to general-purpose robot control: start with a pretrained model, then adapt it to a new robot, task, or environment. The difficulty is that adaptation usually depends on demonstrations collected on the target robot. Physical data collection is slow, expensive, and constrained by access to the hardware, teleoperation systems, and repeated environment resets.

Human demonstrations offer an attractive alternative. People can demonstrate behaviors with relatively lightweight camera equipment, but their bodies, viewpoints, hands, and motion dynamics differ substantially from those of robots. In Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs, Junhyeong Park, Wook Jung, Taeyoung Kim, Huiwon Jang, and Jinwoo Shin introduce H2RB, or Human-To-Robot Bridge, to address both sides of this embodiment gap.

The problem with simply mixing human and robot data

A straightforward co-training strategy combines human and robot demonstrations while giving the model access to each source’s actions. The authors argue that this is not enough. Human and robot observations can look different even when they describe the same task, and the same nominal movement may need to be executed differently by each embodiment.

Prior approaches often place demonstrations into a shared physical action space, such as wrist and hand motion. That can reduce differences in the output representation, but it does not necessarily align the visual features used to predict those actions. It can also force human and robot trajectories through one action mapping, potentially erasing important differences in speed, joint configuration, or execution style.

How H2RB bridges the gap

H2RB uses two complementary mechanisms. The first is geometry-anchored alignment. The method trains a residual adapter on VLA conditioning features so that they encode shared hand geometry: the wrist pose and local fingertip positions. Human geometry is reconstructed from video, while robot geometry comes from observed configurations or kinematics. Because each observation supplies its own geometry target, the method does not require paired human-robot videos or frame-level correspondence.

The second mechanism is a decoupled-decoder latent tokenizer. Human and robot trajectories are first expressed in a common wrist-hand parameterization. A shared temporal encoder then maps both types of action chunks into a latent action space. However, H2RB uses separate human and robot decoders to reconstruct the motions. This lets the latent space capture transferable behavior while allowing each embodiment to preserve its own motion dynamics.

After the tokenizer is trained, its components are frozen. The VLA learns to predict the shared latent action representation with a flow-matching objective, while the geometry adapter is trained alongside it. At deployment, the generated latent is decoded into robot-compatible wrist and hand commands, and the predicted wrist pose is converted into arm motion through inverse kinematics. The authors use the human decoder for their main transfer results because it performs better on the evaluated human-demonstrated tasks.

Results on real robot tasks

The authors evaluate H2RB on an OpenArm humanoid with Inspire hands. The base model is initialized from a robot-pretrained VLA, while target behaviors are introduced through unpaired human demonstrations. They study both object transfer and skill-level transfer.

For object transfer, humans demonstrate pick-and-place tasks involving box, ball, and towel objects that are not present in the relevant robot demonstrations. A matched co-training baseline achieves an average weighted task score of 8.0 out of 100. H2RB raises that score to 51.8. Under shifted object placements, its score is 45.3, compared with 21.3 for the cited shared-action baseline. These scores measure task progress, not just binary completion, so they include stages such as reaching, grasping, lifting, and finishing the placement.

Skill transfer is more demanding: human demonstrations introduce opening and closing an air fryer, behaviors absent from the source robot data. The shared-action baselines score 0.0 on these target skills. H2RB reaches an average score of 62.7 while also reporting 81.0 on the robot’s in-domain pick-and-place evaluation. The authors interpret this as evidence that a common executable action space alone is insufficient for transferring behaviors whose motion distributions and interaction patterns differ substantially.

The paper also reports gains from larger heterogeneous training. Adding multi-embodiment human and robot training increases object-transfer performance from 51.8 to 57.2 and skill-transfer performance from 62.7 to 81.7. An additional instruction-conditioning variant reaches 83.3 on skill transfer, suggesting that repeatedly exposing the action expert to the language instruction can help preserve task semantics. This is a separate enhancement rather than the central H2RB mechanism.

What the analysis suggests

Ablations support the authors’ division of responsibilities between the two bridges. Adding the latent tokenizer improves object transfer from 34.7 to 43.3 and enables a skill-transfer score of 49.7. Adding cross-domain latent regularization further raises the scores to 51.8 and 62.7, while reducing the reported normalized human-robot latent gap from 3.81 to 1.53.

The authors also compare execution speed. A human-transferred policy reaches an average wrist speed of 7.36 centimeters per second, versus 2.52 centimeters per second for a separately trained robot-supervised reference, with similar weighted task scores in that comparison. This suggests that embodiment-specific decoding can preserve useful motion characteristics rather than forcing human demonstrations into the robot policy’s existing dynamics.

Limitations and practical implications

H2RB still depends on several substantial components. Human video must be processed to reconstruct wrist and hand motion, and those estimates must be retargeted to the robot’s coordinate convention. The approach also requires a pretrained robot VLA, a compatible hand-geometry signal, and a tokenizer trained before policy adaptation. These requirements reduce the simplicity of the data-collection story, even if they can be cheaper than gathering extensive target-robot demonstrations.

The evidence is based on real-world evaluations involving particular robot platforms, tasks, demonstrations, and scoring protocols. The paper reports one independently trained policy for each condition and does not establish how well H2RB transfers across much broader variations in camera placement, morphology, object categories, or highly dexterous skills. Its latent alignment regularizes distributions but does not guarantee sample-level semantic correspondence, a limitation the authors explicitly note.

For robotics practitioners, the main lesson is architectural: human data may be most useful when perception and action are aligned without pretending that humans and robots execute movements identically. H2RB indicates that shared geometric references and shared behavioral latents can complement, rather than replace, embodiment-specific control. If its assumptions hold at larger scale, human demonstrations could become a more practical supervision source for adapting pretrained VLAs while reducing the amount of costly robot-specific data required.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close