Generalizable Sparse-View 3D Reconstruction from Unconstrained Images
| Authors | Vinayak Gupta et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.28193 |
| Download | |
| Categories | cs.CV |
Abstract
Reconstructing 3D scenes from sparse, unposed images remains challenging under real-world conditions with varying illumination and transient occlusions. Existing methods rely on scene-specific optimization using appearance embeddings or dynamic masks, which requires extensive per-scene training and fails under sparse views. Moreover, evaluations on limited scenes raise questions about generalization. We present GenWildSplat, a feed-forward framework for sparse-view outdoor reconstruction that requires no per-scene optimization. Given unposed internet images, GenWildSplat predicts depth, camera parameters, and 3D Gaussians in a canonical space using learned geometric priors. An appearance adapter modulates appearance for target lighting conditions, while semantic segmentation handles transient objects. Through curriculum learning on synthetic and real data, GenWildSplat generalizes across diverse illumination and occlusion patterns. Evaluations on PhotoTourism and MegaScenes benchmark demonstrate state-of-the-art feed-forward rendering quality, achieving real-time inference without test-time optimization
Engineering Breakdown
Plain English
GenWildSplat tackles the problem of reconstructing 3D scenes from just a handful of unposed internet images without requiring per-scene optimization, which is a major practical bottleneck in current methods. The system predicts depth maps, camera poses, and 3D Gaussian representations in a canonical space using learned geometric priors, then adapts appearance for different lighting conditions and handles occlusions via semantic segmentation. Unlike existing approaches that need extensive training on each individual scene, GenWildSplat is a feed-forward model that generalizes across scenes and handles real-world challenges like varying illumination and transient objects. This eliminates the expensive per-scene optimization loop that makes current methods impractical for deployed systems.
Core Technical Contribution
The key innovation is moving from scene-specific optimization to a generalizable feed-forward architecture that predicts 3D Gaussians directly from unposed, in-the-wild images without any per-scene training. The authors introduce a learned geometric prior that enables the model to infer depth and camera parameters jointly in a single forward pass, which is a departure from methods that either assume posed images or require extensive optimization. An appearance adapter module decouples geometry from appearance, allowing the same 3D representation to be rendered under different lighting conditions — this is critical for handling the variability in real internet images. Semantic segmentation integration for handling transient occlusions represents a pragmatic engineering choice to deal with real-world messiness that clean datasets ignore.
How It Works
The pipeline begins with a set of unposed RGB images as input and feeds them through a shared feature extraction backbone to build image-level representations. The model then predicts per-image depth maps and camera intrinsics/extrinsics using learned geometric priors that encode knowledge about typical scene structure — this is the core mechanism enabling generalization without per-scene optimization. These predictions are lifted to 3D space where the model learns to represent geometry as 3D Gaussians in a canonical frame, which can be efficiently rendered via rasterization. An appearance adapter network takes the canonical geometry and modulates it for target-specific lighting and viewing conditions, while a semantic segmentation branch identifies transient objects (people, vehicles, etc.) that should be masked or handled separately. Finally, the system can render novel views by rasterizing the adapted Gaussians, enabling both visualization and geometry refinement if needed.
Production Impact
This approach removes a critical bottleneck in 3D reconstruction pipelines: the need to run iterative optimization per-scene, which can take hours or days on GPU clusters for complex scenes. In a production system, you could ingest raw internet images, run a single forward pass on each image set (likely seconds to minutes depending on image count and resolution), and immediately get usable 3D geometry and camera poses without any per-scene tuning. This makes 3D reconstruction accessible for batch processing workflows, real-time applications, and scenarios where you need quick turnaround on diverse unlabeled data. However, the trade-offs include potential accuracy loss compared to heavily optimized per-scene methods, dependency on semantic segmentation quality for handling dynamic objects, and the compute cost of running the feed-forward model at inference time (though likely cheaper than iterative optimization). Integration complexity is moderate — you'd need to implement 3D Gaussian rasterization and the appearance adaptation module, both of which are now standard components in modern graphics pipelines.
Limitations and When Not to Use This
The method assumes that depth, camera pose, and 3D Gaussian predictions from a single forward pass will be sufficiently accurate for practical applications, but this likely underperforms per-scene optimization in cases where fine geometric details matter or scenes have complex non-Lambertian appearance. The reliance on semantic segmentation to handle occlusions means failures propagate — if the segmentation model fails on novel object categories or lighting conditions, the 3D reconstruction degrades. The paper doesn't clarify how the method handles extreme sparsity (e.g., 2-3 images vs. 10+), which matters for real deployment scenarios where image acquisition is costly. Furthermore, the generalization story depends heavily on the diversity and quality of training data, and the abstract doesn't specify what datasets were used or whether the model generalizes to completely different capture environments (e.g., indoor scenes if trained only on outdoor data). The appearance adaptation module assumes lighting changes are the primary source of image variation, which breaks down for scenes with significant material differences or view-dependent effects.
Research Context
This work builds on the recent success of 3D Gaussian splatting as a representation (introduced in 2023) and extends it beyond per-scene optimization toward generalizable reconstruction, similar to how NeRF-based methods evolved from per-scene to generalizable variants. The paper directly addresses limitations of methods like COLMAP + Gaussian splatting pipelines, which require known or recoverable camera poses, by jointly predicting geometry and poses in a feed-forward manner. It represents a convergence of several research threads: learned geometric priors from monocular depth estimation, appearance decoupling from lighting-invariant representation learning, and efficient 3D-aware neural rendering. The generalization focus opens a new research direction toward practical 3D reconstruction tools that don't require the optimization infrastructure of current academic systems, potentially enabling downstream applications in AR/VR, robotics, and autonomous systems that currently struggle with the per-scene optimization bottleneck.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::