3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis
| Authors | Stefan Schulz et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2604.11211 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/
Engineering Breakdown
Plain English
3DTV solves the problem of rendering novel camera views in real-time from sparse multi-camera setups without needing per-scene optimization. The core innovation is a feedforward neural network that takes multiple input camera views and produces high-quality intermediate viewpoints on-the-fly, using geometry-aware depth estimation and intelligent camera selection via Delaunay triangulation. Unlike prior methods that require expensive per-scene fine-tuning, 3DTV generalizes across different scenes with a single trained model, making it practical for AR/VR and telepresence applications where latency matters. The paper demonstrates that this approach matches or exceeds the quality of optimization-based methods while running at interactive frame rates.
Core Technical Contribution
The key innovation is combining lightweight geometric priors (Delaunay-based camera triplet selection and coarse-to-fine depth pyramids) with a feedforward neural network to avoid scene-specific optimization. Prior view synthesis methods required either per-scene neural radiance field optimization (expensive, slow) or relied purely on learned features without geometry (prone to artifacts). 3DTV's pose-aware depth module estimates geometry in a hierarchical manner, enabling efficient feature reprojection and occlusion handling in a single forward pass. The Delaunay triplet selection ensures that each target viewpoint is surrounded by well-distributed source cameras, improving interpolation stability and reducing artifacts from poor camera geometry.
How It Works
The input is a sparse set of camera views (typically 4-8 cameras) with known intrinsics and extrinsics, plus a target camera pose for the novel view to be synthesized. First, the Delaunay-based triplet selector identifies the three source cameras that best surround the target viewpoint in 3D space, ensuring angular coverage. Next, the pose-aware depth module processes the selected source views and the target camera pose to estimate a multi-scale depth pyramid (coarse-to-fine) that represents scene geometry. These depth estimates are used to reproject source image features into the target camera space, and an occlusion-aware blending network learns to combine these reprojected features, handling dis-occlusions and view-dependent effects. Finally, a decoder synthesizes the target view from the blended feature maps. The entire process runs as a single forward pass through the network, with no iterative optimization.
Production Impact
For AR/VR and telepresence systems, this work directly reduces latency to interactive frame rates (30+ fps mentioned implicitly in 'real-time') without requiring expensive per-scene training, which is critical for deployment. Engineers can train a single 3DTV model on a large multi-view video dataset and then deploy it to new scenes instantly with no fine-tuning, drastically simplifying the production pipeline. The geometric priors (Delaunay selection, depth pyramids) make the approach more robust to camera miscalibration and unusual camera configurations compared to purely learned methods. However, the trade-off is that the method still requires known camera parameters and may struggle with extreme baselines or very sparse camera rigs; also, memory usage scales with input resolution and number of source cameras, which impacts mobile deployment.
Limitations and When Not to Use This
3DTV assumes known, accurate camera intrinsics and extrinsics—in practice, calibration errors or dynamic camera rigs will degrade results, and the paper doesn't clearly address how robust the method is to calibration noise. The method is designed for sparse-view interpolation; very dense camera grids or single-image view synthesis remain out of scope, limiting applicability in some capture scenarios. The coarse-to-fine depth estimation may struggle in textureless or highly reflective regions where depth is inherently ambiguous, and the paper provides no analysis of failure modes or per-scene quality metrics. Finally, the Delaunay triplet selection assumes reasonable spatial distribution of cameras; pathological camera configurations (e.g., all cameras in a line) are not discussed, and generalization to scenes with complex occlusions or dynamic content is not thoroughly validated.
Research Context
This work builds on two decades of multi-view rendering research, combining classical computer vision (Delaunay triangulation, epipolar geometry, depth-based reprojection) with modern deep learning. It improves over prior neural view synthesis methods like Neural Radiance Fields (NeRF) and its variants by removing the per-scene optimization bottleneck, and over lightweight interpolation methods by adding learned feature blending and occlusion handling. The paper likely benchmarks on standard multi-view video datasets (e.g., Multiview Extended Video with Alpha, or similar) and establishes a new efficiency-quality frontier for real-time applications. The direction opens up future work on handling dynamic scenes, learning camera selection policies, and integrating global illumination effects.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::