LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis
| Authors | Stanislaw Szymanowicz et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.20176 |
| Download | |
| Categories | cs.CV |
Abstract
Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.
Engineering Breakdown
Plain English
LagerNVS is a neural network that generates novel camera views of 3D scenes in real-time without building explicit 3D models. The key insight is that initializing the encoder with weights from a 3D-supervised reconstruction network, then fine-tuning end-to-end with photometric losses, produces better results than pure 2D-to-2D approaches. The method achieves state-of-the-art performance (31.4 PSNR on Re10k dataset), runs in real-time, generalizes to unconstrained internet images, and can optionally use a diffusion decoder for further refinement. This demonstrates that explicit 3D structure in the learned representations helps downstream view synthesis, even when 3D supervision is removed during the final training phase.
Core Technical Contribution
The paper's central contribution is showing that 3D-aware inductive biases—specifically, pre-training an encoder on explicit 3D reconstruction—provide concrete benefits for novel view synthesis without requiring 3D information at test time. Unlike recent implicit neural rendering methods that learn 3D structure purely from 2D photometric signals, LagerNVS explicitly leverages 3D supervision during encoder initialization, then distills that structure into a lightweight decoder trained end-to-end. This two-stage approach (3D pre-training + end-to-end refinement) is novel because it bridges the gap between fully explicit 3D methods and pure implicit 2D approaches, achieving better generalization and real-time performance than either extreme alone.
How It Works
The pipeline operates in two stages. First, an encoder network is pre-trained on a 3D reconstruction task using ground-truth 3D geometry and camera poses, learning to extract 3D-aware latent features from input images. Second, this pre-trained encoder is frozen or partially tuned, paired with a lightweight decoder, and the combined model is trained end-to-end using photometric losses (pixel-space reconstruction losses comparing rendered outputs to target views) without any 3D supervision. During inference, given one or more input images and optional camera parameters, the encoder produces latent codes that encode 3D structure implicitly, and the decoder generates novel views at arbitrary camera poses. The latent space learned during 3D pre-training acts as an inductive bias—it guides the model toward 3D-geometrically-consistent representations—while the end-to-end photometric training ensures the decoder learns to convert those representations into photorealistic images for any viewpoint.
Production Impact
For engineers building real-world view synthesis systems (e.g., VR/AR applications, 3D asset generation, video interpolation), LagerNVS offers immediate practical wins: real-time inference speed, no need for known camera intrinsics at test time, and strong generalization to uncontrolled imagery. The approach reduces data requirements compared to end-to-end training alone, because the 3D pre-training phase provides a curriculum that accelerates convergence. However, this comes with trade-offs: you need a separate 3D dataset with geometry annotations for the pre-training stage (adding development complexity), and the two-stage pipeline means longer total training time before deployment. The method's compute footprint is modest—it's designed as an encoder-decoder, not a heavy autoregressive model—making it viable for consumer hardware. Integration is straightforward: swap in the pre-trained encoder checkpoint, and the rest of the photometric training loop is standard.
Limitations and When Not to Use This
The paper assumes access to high-quality 3D ground truth for the pre-training stage, which is expensive to obtain and limits the diversity of 3D data that can be used—a serious constraint in production. While the method generalizes to in-the-wild images, the abstract doesn't detail how performance degrades on extreme poses, occlusions, or dynamic scenes, so failure modes in edge cases remain unclear. The claim of real-time rendering is qualified by unspecified resolution and hardware; actual latency budgets for AR/VR applications are not discussed. Finally, the paper's reliance on photometric losses alone may struggle with specular surfaces, transparent materials, or complex lighting, and the optional diffusion decoder adds significant post-hoc inference cost if quality refinement is needed—undermining the real-time claim for critical applications.
Research Context
LagerNVS builds on a decade of work separating explicit 3D reconstruction (Structure-from-Motion, SLAM) from neural rendering (NeRF, neural radiance fields). The key prior insight it leverages is that pre-training on related tasks (here, 3D geometry) improves downstream generalization—similar to how ImageNet pre-training helped vision tasks. The paper improves on recent feed-forward NVS methods (like those using implicit neural representations) by showing that explicit geometric structure during training pays off, even when that structure is discarded at test time. It opens a research direction: how much 3D supervision is 'just right' for learning view synthesis, and whether hybrid explicit-implicit training is a general paradigm for other 3D vision tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::