AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model
| Authors | Yutian Chen et al. |
| Year | 2026 |
| HF Upvotes | 38 |
| arXiv | 2604.19747 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Sparse-view 3D reconstruction is essential for modeling scenes from casual captures, but remain challenging for non-generative reconstruction. Existing diffusion-based approaches mitigates this issues by synthesizing novel views, but they often condition on only one or two capture frames, which restricts geometric consistency and limits scalability to large or diverse scenes. We propose AnyRecon, a scalable framework for reconstruction from arbitrary and unordered sparse inputs that preserves explicit geometric control while supporting flexible conditioning cardinality. To support long-range conditioning, our method constructs a persistent global scene memory via a prepended capture view cache, and removes temporal compression to maintain frame-level correspondence under large viewpoint changes. Beyond better generative model, we also find that the interplay between generation and reconstruction is crucial for large-scale 3D scenes. Thus, we introduce a geometry-aware conditioning strategy that couples generation and reconstruction through an explicit 3D geometric memory and geometry-driven capture-view retrieval. To ensure efficiency, we combine 4-step diffusion distillation with context-window sparse attention to reduce quadratic complexity. Extensive experiments demonstrate robust and scalable reconstruction across irregular inputs, large viewpoint gaps, and long trajectories.
Engineering Breakdown
Plain English
AnyRecon solves the sparse-view 3D reconstruction problem by using a video diffusion model that can condition on arbitrary numbers of unordered input frames rather than just one or two. The key innovation is a persistent global scene memory implemented as a prepended capture view cache that maintains long-range geometric consistency across widely separated viewpoints without temporal compression. This approach scales to large and diverse scenes while preserving explicit geometric control, enabling reconstruction from casual captures with flexible numbers of input views. The method addresses a fundamental limitation of prior diffusion-based approaches that struggled with scalability and geometric consistency when dealing with more than a few conditioning frames.
Core Technical Contribution
The core innovation is the introduction of a persistent global scene memory mechanism—a prepended capture view cache—that enables diffusion models to handle arbitrary-cardinality, unordered sparse inputs while maintaining frame-level correspondence. Unlike prior diffusion-based 3D reconstruction methods that condition on only 1-2 frames (limiting geometric consistency), AnyRecon removes temporal compression and leverages a cache-based architecture to support long-range conditioning across many viewpoints simultaneously. This architectural choice allows the model to scale from a handful of input views to diverse, large-scale scenes without sacrificing geometric fidelity. The explicit geometric control combined with flexible conditioning represents a fundamental shift from single-frame or dual-frame conditioning paradigms that dominated prior work.
How It Works
The system takes arbitrary numbers of unordered sparse input views as conditioning frames and feeds them into a video diffusion model augmented with a persistent global scene memory. The capture view cache is prepended to the input sequence, allowing the model to maintain a consistent reference representation of the scene across all decoder timesteps without losing frame-level correspondence. During the diffusion process, the model iteratively refines novel views and 3D geometry by conditioning on both the cached input views and the noisy intermediate representations from prior diffusion steps. The removal of temporal compression in the cache preserves viewpoint information at full resolution, enabling the model to resolve geometric ambiguities across large viewpoint changes. The output is reconstructed 3D geometry (likely as a neural radiance field, implicit surface, or mesh) along with synthesized novel views that maintain consistency with the input sparse observations. The explicit geometric control—likely through gradient-based optimization or direct geometric constraints—ensures the reconstruction respects camera poses and scene structure rather than relying purely on learned priors.
Production Impact
For production 3D reconstruction pipelines, AnyRecon enables a significant quality upgrade by accepting variable numbers of input views (5 to 20+ frames) rather than requiring carefully curated dual-frame sets, reducing capture logistics complexity on consumer applications and mobile platforms. The scalability to large scenes means reconstruction of entire rooms or outdoor environments becomes feasible without splitting into patches, reducing stitching artifacts and enabling end-to-end optimization. The trade-off is computational cost: video diffusion models with global scene memory are substantially more expensive than feed-forward 3D reconstruction networks, likely requiring GPU inference (5-30 seconds per scene depending on resolution and view count) versus <1 second for lightweight baselines. Integration into production requires managing the cache memory footprint efficiently and handling real-time constraints in interactive applications, though batch reconstruction for asset generation or offline processing becomes straightforward. The flexibility in conditioning cardinality is particularly valuable for e-commerce (variable product photography), virtual reality content creation, and autonomous robotics where input image count varies naturally by scene complexity.
Limitations and When Not to Use This
The paper assumes input views are already roughly aligned or that camera poses can be estimated, which breaks down for severely unstructured captures or scenes with known-view-synthesis failures due to occlusion or repetitive geometry. The removal of temporal compression implies higher memory consumption and computational cost, which may be prohibitive for mobile or edge deployment and for real-time applications requiring <100ms latency. The reliance on diffusion for refinement means inference is inherently slow (multiple denoising steps), and the geometric consistency improvements come at the cost of not being trainable end-to-end with downstream 3D tasks without gradient flow through the diffusion process. The abstract cuts off mid-sentence, suggesting potential limitations around handling extreme viewpoint gaps, dynamic scenes, or thin structures (hair, cloth) are not fully characterized. Follow-up work likely needs to address streaming reconstruction, efficient cache management for very large scenes, and integration with active vision systems that select optimal capture viewpoints.
Research Context
AnyRecon advances the intersection of diffusion models and 3D vision by moving beyond single-image or dual-image priors that dominated prior work like Zero-1-to-3 and similar methods. It builds on foundational work in neural radiance fields (NeRF) and recent breakthroughs in video diffusion models (e.g., Stable Video Diffusion) by combining explicit scene representation with generative priors. The work likely benchmarks on standard sparse-view reconstruction datasets (e.g., DTU, BlendedMVS, or Objaverse) and opens a research direction toward fully flexible multi-view conditioning where view count and ordering become non-factors. This positions AnyRecon as part of a broader shift toward generative priors for 3D reconstruction that can scale beyond niche benchmarks to in-the-wild casual captures.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::