Realizing Immersive Volumetric Video: A Multimodal Framework for 6-DoF VR Engagement
| Authors | Zhengxian Yang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09473 |
| Download | |
| Categories | cs.CV |
Abstract
Fully immersive experiences that tightly integrate 6-DoF visual and auditory interaction are essential for virtual and augmented reality. While such experiences can be achieved through computer-generated content, constructing them directly from real-world captured videos remains largely unexplored. We introduce Immersive Volumetric Videos, a new volumetric media format designed to provide large 6-DoF interaction spaces, audiovisual feedback, and high-resolution, high-frame-rate dynamic content. To support IVV construction, we present ImViD, a multi-view, multi-modal dataset built upon a space-oriented capture philosophy. Our custom capture rig enables synchronized multi-view video-audio acquisition during motion, facilitating efficient capture of complex indoor and outdoor scenes with rich foreground--background interactions and challenging dynamics. The dataset provides 5K-resolution videos at 60 FPS with durations of 1-5 minutes, offering richer spatial, temporal, and multimodal coverage than existing benchmarks. Leveraging this dataset, we develop a dynamic light field reconstruction framework built upon a Gaussian-based spatio-temporal representation, incorporating flow-guided sparse initialization, joint camera temporal calibration, and multi-term spatio-temporal supervision for robust and accurate modeling of complex motion. We further propose, to our knowledge, the first method for sound field reconstruction from such multi-view audiovisual data. Together, these components form a unified pipeline for immersive volumetric video production. Extensive benchmarks and immersive VR experiments demonstrate that our pipeline generates high-quality, temporally stable audiovisual volumetric content with large 6-DoF interaction spaces. This work provides both a foundational definition and a practical construction methodology for immersive volumetric videos.
Engineering Breakdown
Plain English
This paper introduces Immersive Volumetric Videos (IVV), a new media format that captures real-world scenes with full 6-degrees-of-freedom interaction for VR/AR applications, including synchronized video and audio. The authors built ImViD, a multi-view, multi-modal dataset using a custom capture rig that simultaneously records video from multiple camera angles and synchronized audio while capturing dynamic indoor and outdoor scenes. The key innovation is designing a capture pipeline and media format specifically optimized for high-resolution, high-frame-rate content that preserves audiovisual coherence across large interaction spaces, solving the problem that constructing truly immersive experiences from real video—rather than synthetic content—remains largely unexplored in the literature.
Core Technical Contribution
The core technical novelty is the space-oriented capture philosophy and hardware design that enables synchronized multi-view video-audio acquisition during natural motion, combined with a volumetric media format specifically architected for 6-DoF interaction. Unlike prior volumetric video work that typically handles single-view or loosely synchronized multi-modal data, ImViD's capture rig treats the entire spatial environment as the unit of capture rather than individual subjects, enabling coherent audiovisual feedback across large interaction volumes. The dataset design itself is a contribution—it's built from the ground up to support learning-based reconstruction methods that can synthesize novel viewpoints while maintaining audio-visual synchronization and temporal coherence at high frame rates. This represents a shift from treating video and audio as separate modalities to be reconstructed independently, toward building them as intrinsically linked components of immersive volumetric content.
How It Works
The system begins with a custom multi-camera, multi-microphone capture rig that records synchronized video streams from multiple viewpoints and corresponding audio channels during dynamic scene motion. The captured multi-view video-audio data flows into ImViD, which organizes this data around space coordinates rather than individual objects—essentially creating a spatially-indexed volumetric representation where every captured ray and audio sample is tagged with its 6-DoF position and orientation metadata. During processing, the system reconstructs novel viewpoints by interpolating or warping between the multi-view video observations while maintaining temporal coherence frame-to-frame, and crucially, aligning the reconstructed audio to match the synthetic viewpoint's spatial position. The output is a volumetric video representation that supports real-time 6-DoF interactive queries—a user wearing a VR headset can move their head or body through the captured space and see high-resolution video from that new viewpoint with spatially-correct audio, all streamed at high frame rates.
Production Impact
For teams building immersive VR/AR experiences, this approach eliminates the expensive and time-consuming process of manually modeling and animating real-world scenes—you can capture them directly from video. Production pipelines could shift from hiring 3D artists to build synthetic environments toward optimized capture and real-time reconstruction workflows. The main trade-off is massive data and compute requirements: multi-view 4K+ video at 60+ fps with synchronized audio from dozens of cameras generates terabytes of raw data per scene, and real-time 6-DoF view synthesis and audio spatialization demands GPU-accelerated reconstruction pipelines. Integration complexity is significant—you'd need to invest in specialized capture hardware, build custom synchronization infrastructure for video-audio alignment, and develop view-synthesis and audio-rendering components that can run at interactive frame rates on consumer VR hardware, making this accessible primarily to well-resourced studios initially.
Limitations and When Not to Use This
The paper does not address how to handle extremely large-scale scenes or outdoor capture in uncontrolled lighting, and the space-oriented capture philosophy requires dense, synchronized multi-view data that becomes prohibitively expensive to acquire for arbitrarily large environments. The approach assumes that the captured space is reasonably static or has predictable dynamics—highly complex occlusions, transparent surfaces, or specular reflections that break the multi-view geometry assumptions are likely to cause reconstruction artifacts. Audio spatialization in the approach is not deeply detailed in the abstract, leaving open questions about how well spatial audio coherence is maintained during novel-view synthesis, especially with head motion. Additionally, there is no discussion of how to handle variable frame rates across cameras, network-constrained streaming scenarios, or graceful degradation when full 6-DoF data is not available—practical VR systems often need to work on bandwidth-limited headsets.
Research Context
This work builds on decades of volumetric video research and multi-view geometry, but shifts the focus from single-object reconstruction to full-scene immersive capture. It combines insights from the light field and integral imaging literature—which showed that capturing a dense set of rays enables view synthesis—with the practical insights of modern video compression and real-time rendering. The dataset contribution mirrors the role of large-scale benchmarks like ScanNet (3D scene understanding) and Kinetics (action recognition) in enabling new research directions; ImViD provides the foundation for learning-based volumetric video reconstruction at scale. This work opens the research direction of audio-visual scene reconstruction, where audio is not treated as a separate signal but as a spatially-coherent phenomenon that must be synthesized alongside video, which is underexplored relative to vision-only volumetric methods.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::