ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
| Authors | Zihao Huang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.04338 |
| Download | |
| Categories | cs.CV |
Abstract
Synthesizing physically plausible articulated human-object interactions (HOI) without 3D/4D supervision remains a fundamental challenge. While recent zero-shot approaches leverage video diffusion models to synthesize human-object interactions, they are largely confined to rigid-object manipulation and lack explicit 4D geometric reasoning. To bridge this gap, we formulate articulated HOI synthesis as a 4D reconstruction problem from monocular video priors: given only a video generated by a diffusion model, we reconstruct a full 4D articulated scene without any 3D supervision. This reconstruction-based approach treats the generated 2D video as supervision for an inverse rendering problem, recovering geometrically consistent and physically plausible 4D scenes that naturally respect contact, articulation, and temporal coherence. We introduce ArtHOI, the first zero-shot framework for articulated human-object interaction synthesis via 4D reconstruction from video priors. Our key designs are: 1) Flow-based part segmentation: leveraging optical flow as a geometric cue to disentangle dynamic from static regions in monocular video; 2) Decoupled reconstruction pipeline: joint optimization of human motion and object articulation is unstable under monocular ambiguity, so we first recover object articulation, then synthesize human motion conditioned on the reconstructed object states. ArtHOI bridges video-based generation and geometry-aware reconstruction, producing interactions that are both semantically aligned and physically grounded. Across diverse articulated scenes (e.g., opening fridges, cabinets, microwaves), ArtHOI significantly outperforms prior methods in contact accuracy, penetration reduction, and articulation fidelity, extending zero-shot interaction synthesis beyond rigid manipulation through reconstruction-informed synthesis.
Engineering Breakdown
Plain English
This paper tackles synthesizing realistic human-object interactions where objects have articulated parts (like doors, drawers, robotic arms) from just a single video generated by a diffusion model, without requiring any 3D training data. The authors frame this as a 4D reconstruction problem: given a 2D video from a diffusion model as the only supervision signal, they recover a full 3D scene with proper geometry, physics, and contact constraints that evolve over time. This is a significant step beyond prior work that handles only rigid objects and requires explicit 3D labels. The approach inverts the rendering process to extract consistent 4D geometry that respects physical plausibility from purely 2D video priors.
Core Technical Contribution
The core novelty is reformulating articulated human-object interaction synthesis as an inverse rendering optimization problem that works from 2D video supervision alone. Unlike prior zero-shot approaches that generate videos of rigid manipulations without geometric reasoning, this method extracts full 4D scene geometry—including articulated joint configurations, contact points, and temporal consistency—by treating the diffusion-generated video as an inverse rendering target. The key insight is that 2D video frame consistency, optical flow, and contact constraints are sufficient supervision to recover physically plausible 4D scenes without any 3D ground truth. This eliminates the need for expensive 3D/4D datasets while enabling handling of deformable and articulated objects, a class of interactions that prior work largely ignored.
How It Works
The system starts with a video generated by a diffusion model showing a human interacting with an articulated object. The pipeline then sets up an optimization problem where the unknowns are the 4D scene parameters: 3D geometry (mesh or implicit representation), articulated joint states over time, and contact configurations. For each frame in the generated video, the system renders this 4D representation to 2D using differentiable rendering and compares it against the diffusion video. The loss function combines photometric consistency (pixel-level reconstruction), optical flow matching (motion consistency between frames), contact constraints (ensuring objects touch where they should), and physics priors (plausible joint trajectories and forces). The optimization iteratively refines the 4D scene parameters until the rendered output matches the input video while maintaining geometric and physical validity. This inverse rendering approach effectively uses the 2D video as dense supervision without requiring explicit 3D labels.
Production Impact
For teams building human-object interaction systems, this unlocks synthesis of realistic manipulation sequences involving articulated objects without expensive 3D annotation pipelines—a major bottleneck in training data generation. Integration into a production pipeline would replace manual 3D scene setup or synthetic data generation with an automated optimization step: feed diffusion-generated videos through the inverse rendering solver to extract validated 4D scenes that can then be used for downstream tasks like imitation learning, physics simulation, or robotic planning. The trade-off is computational cost: the optimization likely requires 10-100+ GPU seconds per video to converge (typical inverse rendering problems at this scale), so it's suitable for offline dataset generation rather than real-time interaction. The approach also removes dependency on 3D supervision data, reducing the need to maintain expensive capture systems or 3D annotation teams, though you still need a good diffusion model as the video source.
Limitations and When Not to Use This
The approach is fundamentally limited by the quality of the input diffusion video—if the diffusion model generates physically implausible motion or contacts, the inverse rendering will optimize to match these errors rather than correct them, as the 2D video is the only supervision. Articulated objects with complex kinematic chains or many degrees of freedom may be under-constrained by 2D video alone, leading to multiple valid 4D reconstructions; the paper doesn't fully address how to select physically realistic solutions in ambiguous cases. The method assumes access to reasonable object templates or priors to initialize the 4D optimization; completely unknown object categories or unusual articulation structures may fail gracefully. Finally, contact physics in the current formulation appears to handle collision constraints but the paper doesn't describe how well it handles friction, dynamic effects, or deformable contacts—these complex interaction modes likely require additional supervision or modeling.
Research Context
This work builds on the intersection of recent diffusion-based video generation (which has matured to generate compelling human motion) and inverse rendering techniques (widely used in neural radiance field optimization and 3D reconstruction). It extends zero-shot HOI synthesis research that previously handled only rigid objects and rigid deformations by incorporating articulated kinematics and explicit 4D reasoning. The contribution opens up a new direction for supervision-free 3D scene understanding: leveraging generative models as implicit teachers rather than just for final output, which could influence how the community approaches video-to-3D reconstruction more broadly. This also connects to the growing intersection of generative models with geometric reasoning—a promising but still under-explored area for production systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::