LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment
| Authors | Dujun Nie et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2604.11689 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
While the shortage of explicit action data limits Vision-Language-Action (VLA) models, human action videos offer a scalable yet unlabeled data source. A critical challenge in utilizing large-scale human video datasets lies in transforming visual signals into ontology-independent representations, known as latent actions. However, the capacity of latent action representation to derive robust control from visual observations has yet to be rigorously evaluated. We introduce the Latent Action Representation Yielding (LARY) Benchmark, a unified framework for evaluating latent action representations on both high-level semantic actions (what to do) and low-level robotic control (how to do). The comprehensively curated dataset encompasses over one million videos (1,000 hours) spanning 151 action categories, alongside 620K image pairs and 595K motion trajectories across diverse embodiments and environments. Our experiments reveal two crucial insights: (i) General visual foundation models, trained without any action supervision, consistently outperform specialized embodied latent action models. (ii) Latent-based visual space is fundamentally better aligned to physical action space than pixel-based space. These results suggest that general visual representations inherently encode action-relevant knowledge for physical control, and that semantic-level abstraction serves as a fundamentally more effective pathway from vision to action than pixel-level reconstruction.
Engineering Breakdown
Plain English
This paper introduces LARY, a benchmark for evaluating how well latent action representations (learned from unlabeled human videos) can drive robotic control. The core problem is that Vision-Language-Action models need action labels, but scaling these models requires unlabeled video data—so the authors ask: can we learn action representations from raw human videos without explicit annotations, and will those representations transfer to actual robot control? They built a dataset of over one million videos (1,000 hours) across 151 action categories and created a unified evaluation framework that tests both high-level semantic understanding (what action is happening) and low-level robotic execution (precise motor commands). This is the first rigorous benchmark to measure whether unsupervised action representations from human videos can generate robust robot policies.
Core Technical Contribution
The core novelty is the LARY Benchmark itself—a unified, large-scale evaluation framework that bridges the gap between learning from unlabeled human video and actual robotic control. Rather than proposing a new latent action representation method, the authors create the infrastructure to test whether action representations learned from human videos (which are abundant and cheap to collect) can meaningfully drive robotic policies (which require expensive labeled data). The key insight is that latent actions are ontology-independent representations—they don't commit to a specific taxonomy of actions—allowing representations learned from diverse human behavior to transfer to robot tasks. This is methodologically significant because prior work either evaluated latent representations on classification accuracy or on robots in isolation; LARY evaluates end-to-end from human video to robotic control, surfacing whether semantic-level understanding actually translates to low-level control precision.
How It Works
The LARY framework operates in three stages: (1) Representation Learning—human action videos are fed into an encoder (likely a vision-language model backbone) to extract latent action embeddings without task-specific supervision; these embeddings are ontology-independent, meaning they capture action structure without committing to predefined action labels. (2) Evaluation on Semantic Actions—the learned representations are evaluated on a downstream task of identifying what action is occurring (high-level semantic understanding), typically using a linear probe or fine-tuned classifier on top of frozen embeddings. (3) Evaluation on Robotic Control—the same representations are integrated into a visuomotor policy that outputs low-level control commands (joint angles, velocities) given visual observations; this tests whether the semantic structure in the representation actually enables precise motor control. The dataset spans 1,000+ hours and 151 action categories, providing diverse supervisory signal for the semantic evaluation and diverse test scenarios for the robotic control evaluation. The unified framework allows researchers to directly compare how well different latent representation methods perform on both understanding what to do and executing how to do it.
Production Impact
For teams building Vision-Language-Action models or robotic systems, LARY provides a standardized benchmark to validate whether representations learned from cheap, abundant human video data can actually improve robot policies—potentially reducing reliance on costly robot demonstrations. In practice, this means you could pre-train a latent action encoder on massive YouTube-scale human video datasets, then fine-tune downstream robotic policies with far fewer robot trajectories than traditional imitation learning requires. The 1M+ video dataset itself is a production asset; it reduces the barrier to building VLA models since you no longer need to collect 1,000+ hours of annotated robot data to bootstrap representation learning. Trade-offs include: (a) the latent representations may be misaligned with robot embodiment (human arms move differently than robot arms), requiring fine-tuning rather than zero-shot transfer; (b) evaluation requires both semantic task annotation and robot control pipelines, so using LARY in practice means adopting a two-level evaluation regime; (c) the computational cost of encoding 1M videos for baseline comparisons is non-trivial, roughly requiring 100s of GPU hours for a full evaluation pass.
Limitations and When Not to Use This
The paper does not address domain gap—latent actions learned from human behavior may not directly transfer to robot morphologies with different kinematics, payload constraints, or action ranges, so the semantic evaluation results may not predict robotic performance. The 151 action categories, while broad, are still a predefined taxonomy (even if ontology-independent), so the representations may not generalize to novel actions outside this set or to tasks requiring fine-grained temporal precision (e.g., in-hand manipulation). The benchmark assumes that unsupervised representation learning from human video is actually feasible with standard vision encoders; if the visual patterns in human action are fundamentally different from robot camera feeds, even LARY's comprehensive evaluation won't salvage the approach. Follow-up work is needed on: (1) explicit techniques to bridge human-to-robot domain gaps (e.g., adversarial alignment or embodiment-aware pretraining); (2) evaluation on long-horizon tasks requiring temporal reasoning beyond individual frames; (3) testing on robot morphologies and environments not represented in the 151 action set.
Research Context
This work sits at the intersection of vision-language pre-training (like CLIP and its video extensions) and imitation learning from human demonstrations. Prior work on learning from human video either focused on representation quality via classification benchmarks (ignoring downstream control) or tested control on specific domains (without a unified benchmark). LARY builds on foundational ideas in self-supervised learning from video and action recognition but reframes the evaluation to include end-to-end robot policy performance, raising the bar from 'does the representation capture semantics' to 'does it actually enable robot control.' The 1M+ video dataset continues the trend of using internet-scale human video as a pre-training source for robotics, similar to recent work on diffusion-based robot learning from diverse video sources. The benchmark likely catalyzes a new research direction: standardized evaluation of latent action representations, similar to how ImageNet standardized evaluation for visual recognition decades ago.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::