OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
| Authors | Donghao Zhou et al. |
| Year | 2026 |
| HF Upvotes | 69 |
| arXiv | 2604.11804 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.
Engineering Breakdown
Plain English
OmniShow tackles Human-Object Interaction Video Generation (HOIVG), a task that synthesizes realistic videos of people interacting with objects based on multiple input modalities: text descriptions, reference images, audio, and pose sequences. This is practically valuable for e-commerce product demonstrations, short-form video creation, and interactive entertainment platforms where manual video production is expensive and time-consuming. The paper presents an end-to-end framework that unifies all these input conditions into a single coherent video output, addressing a gap where existing methods either ignore some modalities or produce lower-quality results. The authors introduce Unified Channel-wise Conditioning as their core technical contribution to balance controllability (the ability to steer the generation) against output quality.
Core Technical Contribution
The key innovation is Unified Channel-wise Conditioning, a technique that efficiently injects multimodal information (images and poses) into the generation pipeline without the traditional trade-off between control and quality. Prior work either forced users to choose between strong control with mediocre outputs or high-quality results with limited steering capability; this method aims to achieve both simultaneously. The conditioning mechanism appears to operate at the channel level in the underlying diffusion or autoregressive model, allowing fine-grained spatial and temporal control while preserving the model's capacity to generate coherent, high-fidelity content. This represents a genuine architectural contribution rather than just combining existing techniques, as the paper specifically emphasizes the efficiency and harmony of multimodal condition fusion.
How It Works
OmniShow accepts four parallel input streams: text descriptions (what interaction to perform), reference images (visual context and object appearance), audio (timing and emotional tone), and pose sequences (body position and movement). These inputs are processed through separate encoders that extract semantic and spatial features, then unified through the Channel-wise Conditioning mechanism, which distributes this information across the feature channels of the underlying generative model (likely a latent diffusion model or transformer-based video generator). The conditioning is applied at multiple layers and timesteps to ensure temporal consistency across the video frames and coherent interaction dynamics. The model then performs iterative refinement (via diffusion denoising steps or autoregressive generation) to synthesize a video that respects all four modality constraints while maintaining visual quality and realistic physics of human-object interaction. The end-to-end architecture means no intermediate video or keyframe stages are required—the final output directly emerges from the unified conditioning and generation process.
Production Impact
Engineers building video-generation pipelines for e-commerce and social media platforms could dramatically reduce production costs by automating demo video creation: instead of hiring actors and videographers, they upload a product image, write a script (text), add background audio, and provide pose guidance to generate polished interaction videos at scale. The multimodal conditioning means less manual annotation overhead compared to pose-only or text-only systems—you get more expressive control without requiring full motion capture data or extensive manual keyframing. However, there are production trade-offs: the model likely requires significant GPU resources for inference (diffusion models are computationally expensive), each video generation probably takes minutes rather than seconds, and the system likely needs finetuning or retraining when introducing entirely new product categories or interaction types. Integration complexity is moderate—you'd need infrastructure to parse and encode four different input modalities, handle async video processing, and implement fallback behaviors when pose sequences are incomplete or audio is missing.
Limitations and When Not to Use This
The paper does not address extreme edge cases where human-object interaction is physically ambiguous or where multiple valid interpretations exist (e.g., the same pose with an object could mean different interactions in different contexts). It likely assumes that reference images, poses, and text are reasonably well-aligned and that pose sequences are provided in a standard format—handling noisy pose estimation from video or incomplete skeletal data may degrade results significantly. The approach probably struggles with very long videos (beyond a few seconds) due to memory constraints in the generative model, and it may fail to enforce hard physical constraints like preventing interpenetration or maintaining grip correctness throughout. Fine-grained temporal synchronization between audio timing and interaction events is probably not guaranteed, and the model may produce artifacts when audio cues and pose movements are misaligned by even a fraction of a second.
Research Context
This work builds on the broader evolution of multimodal generative models (like DALL-E, Imagen, and video diffusion models) and extends them to the specialized domain of video generation with explicit human dynamics and object interactions. It directly advances research on video understanding and synthesis by treating pose as a first-class conditioning signal alongside text and images, rather than a secondary constraint. OmniShow likely improves upon benchmarks in video generation quality metrics (LPIPS, FVD, or similar) while also introducing new evaluation criteria specific to interaction coherence and object manipulation realism. The work opens research directions in physics-aware video generation, cross-modal alignment in long-form video synthesis, and practical scalability of multimodal diffusion models for content creation workflows.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::