Lighting-grounded Video Generation with Renderer-based Agent Reasoning
| Authors | Ziqi Cai et al. |
| Year | 2026 |
| HF Upvotes | 7 |
| arXiv | 2604.07966 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Diffusion models have achieved remarkable progress in video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera trajectory are often entangled or only weakly modeled, restricting their applicability in domains like filmmaking and virtual production where explicit scene control is essential. We present LiVER, a diffusion-based framework for scene-controllable video generation. To achieve this, we introduce a novel framework that conditions video synthesis on explicit 3D scene properties, supported by a new large-scale dataset with dense annotations of object layout, lighting, and camera parameters. Our method disentangles these properties by rendering control signals from a unified 3D representation. We propose a lightweight conditioning module and a progressive training strategy to integrate these signals into a foundational video diffusion model, ensuring stable convergence and high fidelity. Our framework enables a wide range of applications, including image-to-video and video-to-video synthesis where the underlying 3D scene is fully editable. To further enhance usability, we develop a scene agent that automatically translates high-level user instructions into the required 3D control signals. Experiments show that LiVER achieves state-of-the-art photorealism and temporal consistency while enabling precise, disentangled control over scene factors, setting a new standard for controllable video generation.
Engineering Breakdown
Plain English
LiVER is a video generation system built on diffusion models that lets you control exactly how scenes look by specifying 3D properties like object positions, lighting, and camera movement. Current video diffusion models treat these factors as entangled, making it hard to get precise control in professional applications like filmmaking and VFX. The authors introduce a rendering-based conditioning approach that decouples lighting, layout, and camera parameters into separate control signals derived from a unified 3D representation. They also created a large-scale annotated dataset with dense labels for these scene properties, enabling the model to learn and respect explicit scene control during generation.
Core Technical Contribution
The key innovation is using a 3D renderer as an intermediary to ground video diffusion models in explicit scene properties rather than trying to learn these factors implicitly from unstructured data. Instead of feeding raw image inputs to the diffusion model, the system renders control signals (lighting maps, depth from layout, camera projection matrices) from a canonical 3D scene representation and conditions the model on these structured signals. This disentanglement approach means you can modify any scene property independently—change lighting without moving objects, or adjust camera without affecting lighting—something prior video diffusion models cannot reliably do. The framework is intentionally lightweight, suggesting efficient inference suitable for iterative production workflows where artists need quick feedback.
How It Works
The system takes as input a 3D scene representation (object positions, geometry, material properties, light sources) plus desired video parameters (frame count, resolution, desired camera trajectory). A differentiable renderer generates conditioning signals: spatial layout maps showing object positions and boundaries, lighting render passes decomposing direct and indirect illumination, and camera parameter grids encoding viewpoint information. These rendered signals are encoded into embedding space and injected into the diffusion model via cross-attention layers at multiple scales. During the reverse diffusion process (noise-to-video generation), the model uses these structured conditioning signals to synthesize video frames that respect the specified 3D scene properties. The tight integration with rendering means changes to scene properties automatically propagate to the generated video without retraining.
Production Impact
This directly addresses a pain point in VFX and virtual production pipelines: today's video generation tools require expensive trial-and-error iteration because you cannot control specific scene aspects. With LiVER, a VFX artist could specify exact lighting setups, actor positions, and camera moves, then generate consistent video—reducing the number of manual takes or re-renders needed. Integration into production requires interfacing with existing 3D engines (Unreal, Maya) to export scene representations and camera parameters; this is straightforward since the system uses standard 3D formats. Trade-offs include the need for dense annotated training data (the authors created this, but similar datasets may not exist for specialized domains) and the computational cost of the differentiable renderer during inference, which adds latency beyond standard diffusion sampling. For teams with controlled environments (studio sets, fixed lighting rigs), this approach could reduce post-production costs significantly by enabling one-shot generation with correct scene properties rather than shooting multiple takes.
Limitations and When Not to Use This
The method's effectiveness depends heavily on accurate 3D scene representation and renderer fidelity—if the 3D model doesn't match reality, the generated video inherits those errors, creating a garbage-in-garbage-out failure mode. The approach assumes scene properties can be cleanly decomposed into layout, lighting, and camera parameters, which breaks down for complex effects like volumetric fog, subsurface scattering, or complex material interactions that don't factor neatly. The large-scale dataset requirement means this likely doesn't generalize to highly specialized domains (e.g., underwater environments, extreme weather) unless equivalent annotated data exists. The paper's abstract is incomplete (cuts off mid-sentence about the lightweight component), suggesting details on inference speed, memory footprint, and scalability to high-resolution video are missing from the provided text.
Research Context
This work builds on the diffusion model foundation that has dominated image and video generation since 2020, but shifts from purely learned representations toward explicit 3D grounding—a trend also seen in neural radiance fields and 3D-aware generative models. It advances beyond prior conditional diffusion methods (like ControlNet) by recognizing that video-specific factors like lighting and camera need domain-specific rendering-based conditioning rather than generic spatial controls. The contribution of a large-scale densely-annotated video dataset addresses a longstanding bottleneck in video generation research, similar to how COCO and Kinetics enabled progress in their respective domains. This opens a research direction in renderer-guided generation: leveraging inverse graphics and differentiable rendering as conditioning mechanisms for generative models, which could extend to 3D synthesis, scene editing, and photorealistic animation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::