ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
| Authors | Jindi Lv et al. |
| Year | 2026 |
| HF Upvotes | 15 |
| arXiv | 2604.08168 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Vision-language-action (VLA) models have advanced robot manipulation through large-scale pretraining, but real-world deployment remains challenging due to partial observability and delayed feedback. Reinforcement learning addresses this via value functions, which assess task progress and guide policy improvement. However, existing value models built on vision-language models (VLMs) struggle to capture temporal dynamics, undermining reliable value estimation in long-horizon tasks. In this paper, we propose ViVa, a video-generative value model that repurposes a pretrained video generator for value estimation. Taking the current observation and robot proprioception as input, ViVa jointly predicts future proprioception and a scalar value for the current state. By leveraging the spatiotemporal priors of a pretrained video generator, our approach grounds value estimation in anticipated embodiment dynamics, moving beyond static snapshots to intrinsically couple value with foresight. Integrated into RECAP, ViVa delivers substantial improvements on real-world box assembly. Qualitative analysis across all three tasks confirms that ViVa produces more reliable value signals, accurately reflecting task progress. By leveraging spatiotemporal priors from video corpora, ViVa also generalizes to novel objects, highlighting the promise of video-generative models for value estimation.
Engineering Breakdown
Plain English
ViVa addresses a critical gap in robot reinforcement learning: existing vision-language models (VLMs) can predict robot actions but struggle to assess how well a state will lead to task completion, especially in long-horizon manipulation tasks with partial observability and delayed rewards. The paper proposes ViVa, which repurposes a pretrained video generator to jointly predict future robot proprioception (joint angles, velocities) and a scalar value estimate for the current state. By leveraging the spatiotemporal understanding already learned by video generators, ViVa provides more reliable value estimates than traditional value models built on frozen VLMs, enabling better reinforcement learning for real-world robot control.
Core Technical Contribution
The core novelty is reframing video generation as a value estimation task. Rather than training a separate value head on top of a vision-language model, ViVa uses a pretrained video diffusion or autoregressive model as its backbone and adds a value prediction head that operates in the latent space of video generation. This exploits the fact that good video prediction requires understanding state dynamics and task progress implicitly—a video generator that can forecast future proprioception has already learned to model task-relevant temporal patterns. The key insight is that the spatiotemporal representational capacity of video models, trained on diverse real-world video data, transfers better to value estimation than static VLM representations, especially for capturing the dynamics needed in long-horizon tasks.
How It Works
ViVa takes as input the current RGB observation (from robot camera) and the robot's proprioceptive state (joint positions, velocities, end-effector pose). These are encoded into the latent space of a pretrained video generator (e.g., a model like Latent Video Diffusion or similar). The model then operates in two branches: (1) a video generation branch that autoregressively or diffusion-based predicts the next K frames of proprioception in latent space, and (2) a value head that consumes the same latent representation and outputs a scalar Q-value or state value estimate. During training, the video prediction loss (reconstruction of ground-truth future proprioception) and the value loss (temporal difference error or other RL objective) are jointly optimized, allowing the video decoder to learn task-relevant dynamics while the value head learns to assess progress. During inference, the value estimate is extracted from the model's latent representation without requiring full video decoding, making it efficient for real-time RL rollouts.
Production Impact
In production robot systems, ViVa would replace the typical pipeline of (frozen pretrained VLM + learned value head) with a single video-generative model that provides both action predictions and value estimates. This reduces model complexity—one pretrained backbone instead of multiple specialized models—and improves value estimation reliability in partially observable environments, which directly translates to faster policy learning and more stable exploration during real-world robot training. The main trade-off is computational cost: video generation models are larger and slower than static VLM embeddings, so inference latency for value estimation may increase by 2–5× unless the latent video space is very efficient; you'd need to profile on your target hardware (edge GPUs, robot computers). Integration complexity is moderate: you need a pretrained video model (not all roboticists have access to large-scale video diffusion models), and you must handle the joint training objective carefully to avoid one loss dominating the other. For tasks with long horizons (15+ steps) or heavy partial observability, the improved value estimates likely justify the computational overhead.
Limitations and When Not to Use This
ViVa assumes access to a high-quality pretrained video model, which is a strong assumption if you're working with specialized robot domains where general video data may not transfer well; the paper doesn't thoroughly evaluate transfer from generic video models to niche robot tasks (surgery, industrial manipulation with unusual visual appearance). The joint training of video prediction and value estimation introduces a multi-task learning balancing problem—if not tuned correctly, the video loss can dominate and the value estimates may be noisy, or vice versa; the paper likely needs more ablation on loss weighting. The approach is evaluated primarily on simulation or controlled lab settings; real-world deployment with visual distractors, occlusions, and distribution shift may expose failure modes where the video predictor's assumptions break down. Additionally, the method doesn't inherently solve the credit assignment problem in very long-horizon tasks; value estimates are still only as good as the RL training objective, so tasks with sparse rewards still require careful reward shaping or other credit assignment mechanisms.
Research Context
This work builds on the recent success of Vision-Language-Action (VLA) models like RT-1, Diffusion Policy, and others that leverage large-scale pretraining for robot learning. It also builds on the growing literature of using video diffusion models (e.g., Latent Video Diffusion, Make-a-Video) as general-purpose spatiotemporal encoders. The paper is motivated by the well-known limitation that frozen VLM representations are static and don't capture temporal dynamics, which prior work (e.g., on Dreamer and world models) addressed via explicit dynamics modeling. ViVa's contribution is to show that video generation, which is already trained to model dynamics, can serve double-duty as a value estimator, improving data efficiency and value estimation reliability. This opens a research direction: using generative models (video, action-conditional prediction, etc.) as dual-purpose backbones for both planning and value estimation, potentially reducing the number of specialized components in RL systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::