CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation
| Authors | Xiangyang Luo et al. |
| Year | 2026 |
| HF Upvotes | 85 |
| arXiv | 2604.19636 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.
Engineering Breakdown
Plain English
CoInteract is a framework for generating realistic videos of people interacting with products, conditioned on a person image, product image, text description, and speech audio. The core problem it solves is that existing diffusion models struggle with two critical failure modes: maintaining structural stability in sensitive areas like hands and faces, and ensuring physically plausible contact (like preventing hands from passing through objects). The paper proposes a Human-Aware Mixture-of-Experts routing mechanism embedded in a Diffusion Transformer backbone to address these specific constraints. This work targets practical applications in e-commerce and digital advertising where realistic product interaction videos are needed at scale.
Core Technical Contribution
The technical novelty centers on a Human-Aware Mixture-of-Experts (MoE) module that routes different tokens to lightweight, region-specialized expert networks within a Diffusion Transformer. This is fundamentally different from prior approaches because it explicitly recognizes that human-object interaction synthesis requires specialized handling for anatomically sensitive regions (hands, faces) and physical contact constraints, rather than treating the entire generation space uniformly. The MoE architecture allows the model to learn separate, optimized representations for human body parts versus object regions versus interaction zones, enabling better structural stability and contact plausibility simultaneously. By combining region-specialized experts with a diffusion-based generative approach, CoInteract achieves a better balance between photorealistic rendering and physical constraint satisfaction than monolithic diffusion models.
How It Works
The pipeline accepts four conditioning inputs: a person reference image, a product reference image, text descriptions, and speech audio. These are encoded into embeddings and injected into a Diffusion Transformer backbone that iteratively denoises a random video tensor into coherent frames. The Human-Aware MoE module intercepts token embeddings at multiple layers and routes each token to one of several lightweight expert networks based on whether it corresponds to human body regions, object regions, or interaction contact zones. Each expert is trained to handle its specialized region with appropriate inductive biases—for example, hand experts learn to maintain anatomical consistency and prevent self-intersection, while contact experts explicitly model hand-object surface relationships. The diffusion process runs for multiple timesteps, progressively refining the video frames while respecting both the conditioning signals and the region-specific constraints enforced by the MoE routing. Output is a multi-frame video sequence showing realistic human-product interaction.
Production Impact
In production e-commerce and advertising systems, this approach would eliminate costly manual video production for product demonstrations, enabling dynamic generation of interaction videos tailored to different demographics, products, and marketing contexts. Integration would require: (1) a video generation service accepting multimodal conditioning (image + text + audio), (2) prompt engineering pipelines for consistent product-person pairing descriptions, and (3) postprocessing for temporal coherence across long sequences. The main trade-off is computational cost—Diffusion Transformers with MoE routing require significant GPU memory and inference time per video (likely minutes per 10-second clip), so you'd need dedicated inference infrastructure or batch processing for high volume. The region-specialized MoE adds model complexity and training overhead compared to baseline diffusion models, but this is justified if your application specifically requires hands-object contact plausibility (e-commerce, AR try-on, virtual influencer content). Latency would be a concern for real-time applications; this is better suited for batch generation or pre-production workflows.
Limitations and When Not to Use This
The paper's scope is limited to human-object interaction synthesis and doesn't address multi-person scenarios, which are common in real e-commerce environments (e.g., group product demonstrations). The approach assumes high-quality reference images and detailed text prompts are available; performance likely degrades significantly with low-resolution, blurry, or ambiguous conditioning inputs, which are common in production datasets. The physical plausibility constraints (avoiding interpenetration) are learned implicitly through the MoE routing, but there's no guarantee the model won't violate these constraints in novel interaction scenarios it hasn't seen during training—you'd still need post-hoc validation or human review for critical applications. The dependence on speech audio for temporal alignment adds a constraint; many e-commerce videos may not have well-synchronized speech, limiting the framework's applicability. Finally, the paper doesn't provide detailed ablation studies on the MoE architecture or quantitative metrics comparing contact plausibility against baselines, making it difficult to assess exactly which components drive the improvements.
Research Context
This work builds on the recent surge in diffusion-based video generation (e.g., Imagen Video, Make-A-Video, Runway) while addressing a specific pain point those models don't handle well—physical interaction constraints. It extends the Mixture-of-Experts concept (popularized in models like Switch Transformers and large language models) into the spatial/anatomical domain, routing tokens based on semantic region identity rather than expert load balancing. The paper advances the intersection of conditional video generation and physical constraint satisfaction, opening up applications in embodied AI, robotics simulation, and VR content creation. By combining region-specialized experts with diffusion, it provides a blueprint for other constrained generation problems (e.g., hand-tool interaction in manufacturing, sports motion synthesis) where certain regions have strict physical or anatomical requirements.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::