DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation
| Authors | Hyeonwoo Kim et al. |
| Year | 2026 |
| HF Upvotes | 24 |
| arXiv | 2604.20841 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent advances in video generative models enable the synthesis of realistic human-object interaction videos across a wide range of scenarios and object categories, including complex dexterous manipulations that are difficult to capture with motion capture systems. While the rich interaction knowledge embedded in these synthetic videos holds strong potential for motion planning in dexterous robotic manipulation, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets in physics-based character control. We present DeVI (Dexterous Video Imitation), a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects. To overcome the imprecision of generative 2D cues, we introduce a hybrid tracking reward that integrates 3D human tracking with robust 2D object tracking. Unlike methods relying on high-quality 3D kinematic demonstrations, DeVI requires only the generated video, enabling zero-shot generalization across diverse objects and interaction types. Extensive experiments demonstrate that DeVI outperforms existing approaches that imitate 3D human-object interaction demonstrations, particularly in modeling dexterous hand-object interactions. We further validate the effectiveness of DeVI in multi-object scenes and text-driven action diversity, showcasing the advantage of using video as an HOI-aware motion planner.
Engineering Breakdown
Plain English
This paper presents DeVI, a framework that converts text-conditioned synthetic videos from generative models into physically plausible robot control policies for dexterous manipulation tasks. The key innovation is bridging the gap between 2D video synthesis (which is unrealistic for physics simulation) and 3D physics-based robot control by extracting motion knowledge from videos and refining it through physics constraints. The authors demonstrate that their approach enables robots to perform complex dexterous interactions with previously unseen objects, solving a critical problem in imitation learning where motion capture data is scarce or impossible to collect for certain manipulation skills.
Core Technical Contribution
The core contribution is a novel pipeline that uses video generative models as a source of interaction priors rather than direct imitation targets. DeVI overcomes the 2D-to-3D gap and physical fidelity limitations by combining video feature extraction with physics-based trajectory optimization and refinement. The framework is conditioned on natural language descriptions, enabling flexible task specification without paired motion-video datasets. This represents a fundamental shift in how synthetic visual data can inform robot learning—treating videos as semantic guidance rather than pixel-perfect reference trajectories.
How It Works
The system ingests text-conditioned synthetic videos generated by recent diffusion or transformer-based video models, extracting high-level interaction semantics and spatial-temporal structure. These video features are then used as guidance for a physics-based character controller that generates 3D joint trajectories compatible with rigid-body dynamics simulation. The pipeline likely includes a perception module mapping video frames to object/hand pose estimates, a trajectory optimization layer that satisfies physics constraints while staying faithful to video guidance, and a refinement stage using physics simulation feedback. The final output is a control policy (joint angles, torques, or contact forces over time) that can be executed on real robots while maintaining the interaction intent from the video.
Production Impact
For robotics teams building dexterous manipulation systems, DeVI eliminates dependency on expensive motion capture infrastructure or hand-crafted policies for novel object interactions. This directly reduces the cost and timeline for scaling manipulation skills to new object categories—instead of weeks of mocap sessions or manual programming, engineers can generate task specifications in natural language and iterate on the learned behavior. The physics-based refinement stage ensures sim-to-real transfer is more tractable since trajectories already satisfy dynamics constraints. Trade-offs include moderate compute overhead for trajectory optimization (likely seconds to minutes per task) and the requirement to maintain accurate 3D simulators and physics parameters for the target robot platform. Integration requires a vision system for real-time object detection/pose estimation and careful validation that synthetic video semantics transfer to real-world object variations.
Limitations and When Not to Use This
The framework assumes accurate 3D object pose estimation and physics simulation of the target environment, which can be failure points in real-world deployment with occlusions or deformable objects. The approach is fundamentally limited by the quality and diversity of the underlying video generative model—if the model cannot synthesize certain interaction types, DeVI cannot learn them. The paper does not address failure recovery or reactive control, meaning the learned policies are open-loop trajectory execution without online adaptation to disturbances. Additionally, the method likely struggles with contact-rich manipulation where precise force/contact timing is critical, since video supervision provides only visual evidence of contact without explicit force labels. Real-world validation on a full range of object categories and manipulation complexities is still needed, and computational cost during policy learning may limit rapid iteration in practice.
Research Context
This work builds on recent advances in text-to-video generation (Sora, Runway, etc.) and physics-informed machine learning, addressing a well-known bottleneck in learning-based robotics: the scarcity of diverse, realistic manipulation demonstrations. It extends prior work in imitation learning from video (e.g., behavioral cloning from visual observations) by adding explicit physics constraints and leveraging generative models as data sources rather than passive recorders. The research aligns with emerging directions in using large generative models as world models and prior knowledge sources for downstream control tasks. This opens future work on scaling to full-body humanoid control, multi-agent coordination from synthetic videos, and closed-loop policy learning with video-based reward functions.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::