ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration
| Authors | Fengyuan Yang et al. |
| Year | 2026 |
| HF Upvotes | 13 |
| arXiv | 2604.01043 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent advances in Video Foundation Models (VFMs) have revolutionized human-centric video synthesis, yet fine-grained and independent editing of subjects and scenes remains a critical challenge. Recent attempts to incorporate richer environment control through rigid 3D geometric compositions often encounter a stark trade-off between precise control and generative flexibility. Furthermore, the heavy 3D pre-processing still limits practical scalability. In this paper, we propose ONE-SHOT, a parameter-efficient framework for compositional human-environment video generation. Our key insight is to factorize the generative process into disentangled signals. Specifically, we introduce a canonical-space injection mechanism that decouples human dynamics from environmental cues via cross-attention. We also propose Dynamic-Grounded-RoPE, a novel positional embedding strategy that establishes spatial correspondences between disparate spatial domains without any heuristic 3D alignments. To support long-horizon synthesis, we introduce a Hybrid Context Integration mechanism to maintain subject and scene consistency across minute-level generations. Experiments demonstrate that our method significantly outperforms state-of-the-art methods, offering superior structural control and creative diversity for video synthesis. Our project has been available on: https://martayang.github.io/ONE-SHOT/.
Engineering Breakdown
Plain English
This paper introduces ONE-SHOT, a framework that enables fine-grained editing of both people and environments in video synthesis, addressing a major limitation of current video foundation models. The authors solve a critical trade-off problem: existing approaches require heavy 3D preprocessing to gain precise environmental control, which sacrifices the generative flexibility that makes foundation models powerful. Their key innovation is a canonical-space injection mechanism that uses cross-attention to cleanly separate human motion dynamics from scene/environment cues, enabling independent control of each without pre-computed 3D geometry. This parameter-efficient approach makes compositional video generation practical and scalable.
Core Technical Contribution
The core novelty is the canonical-space injection mechanism that factorizes video generation into disentangled signals for humans and environments. Unlike prior work that relies on rigid 3D geometric compositions (which require expensive preprocessing and limit flexibility), this approach uses cross-attention within the video foundation model's latent space to decouple these two signal types. The framework is parameter-efficient, meaning it adds minimal trainable parameters to the base VFM rather than requiring full model retraining. This represents a fundamental shift from explicit geometric control to learned, attention-based factorization that preserves the generative capabilities of foundation models.
How It Works
ONE-SHOT operates by first taking a video foundation model and augmenting it with a canonical-space injection module positioned at cross-attention layers. When generating a video, the system receives dual inputs: (1) human-centric control signals describing subject pose, motion, or action, and (2) environment-centric signals describing scene layout, lighting, or background. The canonical-space injection intercepts cross-attention computations and routes human motion signals to attend over representations of the human subject while routing environment signals to attend over scene representations, effectively creating separate information pathways. This disentanglement happens within the model's forward pass without requiring explicit 3D mesh reconstruction or geometric alignment. The output is a synthesized video where humans and scenes can be controlled independently while maintaining coherent interaction and photorealism.
Production Impact
For teams building video editing or synthesis products, this approach dramatically reduces the engineering overhead compared to 3D-aware baselines—no need to reconstruct 3D scenes, register meshes, or maintain complex geometric pipelines. The parameter-efficiency (likely LoRA-style adapters rather than full fine-tuning) means deployment footprint is small and inference latency should be comparable to the base VFM. Production pipelines could accept simple control inputs (e.g., pose keyframes for humans, scene descriptions for environments) without requiring users to pre-process 3D assets. The main trade-off is that control precision may not match custom 3D-aware systems in edge cases where exact geometric relationships matter critically, and the approach's effectiveness depends on having a sufficiently capable base VFM. Integration would involve wrapping the base VFM with the canonical-space injection module and building UI/APIs to accept separated human and environment control signals.
Limitations and When Not to Use This
The paper's reliance on video foundation models means it inherits those models' limitations—if the base VFM struggles with a particular visual domain, ONE-SHOT will too, and the injection mechanism cannot fully compensate. The disentanglement via cross-attention is learned rather than explicitly constrained, so in practice there may be residual entanglement where human motion inadvertently affects background synthesis or vice versa, especially in complex scenes. The approach assumes human and environment signals can be cleanly separated at the control input level, which may be unrealistic for videos where human action fundamentally reshapes the scene (e.g., moving objects, activating lights). The paper does not clearly specify what happens when human dynamics and environment control conflict (e.g., a person walking into a wall), suggesting failure modes remain unaddressed in scenarios requiring strong physical consistency.
Research Context
This work builds on the recent surge in video foundation models (like Stable Video Diffusion, Make-A-Video, and similar large-scale diffusion-based approaches) that have shown impressive generative capabilities but lack fine-grained compositional control. It directly addresses a known bottleneck: while image-based foundation models have benefited from adapter-based control mechanisms (ControlNet, spatial guidance), video models have lagged because temporal coherence makes it harder to impose explicit constraints without breaking the generative process. The paper's canonical-space factorization approach echoes prior work in disentangled representation learning (e.g., β-VAE, Factor-VAE) but applies it in the cross-attention layer of foundation models, which is a novel architectural location. This opens a research direction toward learning-based, attention-mediated control for multimodal generation, potentially applicable beyond video to 3D scene synthesis and interactive environments.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::