Action Images: End-to-End Policy Learning via Multiview Video Generation
| Authors | Haoyu Zhen et al. |
| Year | 2026 |
| HF Upvotes | 12 |
| arXiv | 2604.06168 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
World action models (WAMs) have emerged as a promising direction for robot policy learning, as they can leverage powerful video backbones to model the future states. However, existing approaches often rely on separate action modules, or use action representations that are not pixel-grounded, making it difficult to fully exploit the pretrained knowledge of video models and limiting transfer across viewpoints and environments. In this work, we present Action Images, a unified world action model that formulates policy learning as multiview video generation. Instead of encoding control as low-dimensional tokens, we translate 7-DoF robot actions into interpretable action images: multi-view action videos that are grounded in 2D pixels and explicitly track robot-arm motion. This pixel-grounded action representation allows the video backbone itself to act as a zero-shot policy, without a separate policy head or action module. Beyond control, the same unified model supports video-action joint generation, action-conditioned video generation, and action labeling under a shared representation. On RLBench and real-world evaluations, our model achieves the strongest zero-shot success rates and improves video-action joint generation quality over prior video-space world models, suggesting that interpretable action images are a promising route to policy learning.
Engineering Breakdown
Plain English
This paper introduces Action Images, a new approach to robot policy learning that treats action specification as a video generation problem. Instead of encoding robot actions as low-dimensional tokens (the standard approach), the authors convert 7-DoF robot arm motions into interpretable multi-view action videos that are grounded in actual pixel coordinates. This allows the system to leverage powerful pretrained video models while maintaining clear visual grounding of what action the robot should take. The key innovation is unifying world action modeling and policy learning through pixel-space action representations, which improves transfer across different viewpoints and environments compared to existing methods.
Core Technical Contribution
The core novelty is reframing robot action representation as pixel-grounded action images—essentially generating multi-view video sequences that explicitly visualize robot-arm trajectories rather than encoding actions as discrete tokens or latent vectors. This pixel-grounding approach allows the method to directly exploit pretrained video backbones designed for natural video understanding, inheriting their rich visual priors without requiring separate action-specific modules. By making actions transparent and viewpoint-agnostic through multi-view generation, the approach naturally enables generalization across camera angles and environment variations without explicit view-translation machinery. The unified formulation of policy learning as video generation is fundamentally different from prior world action models that treat action as a separate encoding problem orthogonal to state prediction.
How It Works
The system takes a reference image (current robot state), a target action specification, and generates multi-view action videos showing how the robot arm should move to execute that action. The input action (7-DoF end-effector target or trajectory) is converted into action images—pixel-space visualizations of the robot's planned motion—by rendering predicted robot poses across multiple camera viewpoints. These action images are then fed into a pretrained video generation backbone (likely a diffusion or transformer-based model) alongside the current state observation to predict future frames showing the action execution. The architecture leverages the video model's temporal and spatial understanding to implicitly learn the dynamics consequences of actions while maintaining pixel-level interpretability. During policy learning, the model can be trained end-to-end to predict action images that, when executed, lead to desired state transitions, effectively distilling policy knowledge into the video generation process.
Production Impact
Adopting this approach would fundamentally change how you represent and communicate robot actions in production systems. Instead of managing separate token vocabularies or latent action spaces, you'd work with visually interpretable action images that can be debugged by human operators—you can literally see what motion the policy intends before execution. This enables stronger transfer to new robot embodiments and camera configurations because the pixel-space grounding naturally handles viewpoint variation without retraining. The computational cost is higher than token-based methods (video generation is expensive), but the ability to leverage massive pretrained video models could offset this through better sample efficiency and generalization, reducing the real-world robot data collection burden. Integration complexity increases because you need rendering pipelines to convert 7-DoF actions to multi-view visualizations, but this becomes a one-time infrastructure investment that then applies to any downstream policy learning task.
Limitations and When Not to Use This
The paper's approach assumes you have accurate forward models or rendering engines to convert robot actions into pixel-space action images, which requires precise pose estimation and kinematics—this breaks down with articulated objects or contact-rich manipulation where pixel-level grounding becomes ambiguous. Generating high-quality multi-view videos at inference time is computationally expensive, potentially making real-time policy execution latency-prohibitive for reactive tasks (the paper doesn't discuss actual wall-clock deployment times). The method inherits all limitations of the underlying video generation model, including potential hallucination of physically impossible motions or slow adaptation to novel object properties or surface friction that weren't represented in training video data. The paper doesn't address how to handle actions with multiple valid solutions (e.g., grasping an object from different angles), where pixel-space grounding could over-constrain the policy unnecessarily.
Research Context
This work builds on the emerging world action model paradigm (following work by Dreamer and subsequent papers) that unifies state and action modeling within a single learned environment model, but pushes toward pixel-grounded representations rather than latent token bottlenecks. It addresses a known weakness in prior WAM approaches: the difficulty of transferring knowledge from massive pretrained video models because actions were encoded separately from state, preventing the video backbone from learning action-conditioned dynamics natively. The multi-view generation aspect draws from recent progress in 3D-aware video synthesis and view-consistent generation, applying those techniques to make robot actions viewpoint-invariant. This opens a research direction toward fully pixel-grounded policy learning where every component (state, action, dynamics) remains interpretable and leverages the exponential scaling of vision-language and video foundation models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::