UFO-4D: Unposed Feedforward 4D Reconstruction from Two Images
| Authors | Junhwa Hur et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24290 |
| Download | |
| Categories | cs.CV |
Abstract
Dense 4D reconstruction from unposed images remains a critical challenge, with current methods relying on slow test-time optimization or fragmented, task-specific feedforward models. We introduce UFO-4D, a unified feedforward framework to reconstruct a dense, explicit 4D representation from just a pair of unposed images. UFO-4D directly estimates dynamic 3D Gaussian Splats, enabling the joint and consistent estimation of 3D geometry, 3D motion, and camera pose in a feedforward manner. Our core insight is that differentiably rendering multiple signals from a single Dynamic 3D Gaussian representation offers major training advantages. This approach enables a self-supervised image synthesis loss while tightly coupling appearance, depth, and motion. Since all modalities share the same geometric primitives, supervising one inherently regularizes and improves the others. This synergy overcomes data scarcity, allowing UFO-4D to outperform prior work by up to 3 times in joint geometry, motion, and camera pose estimation. Our representation also enables high-fidelity 4D interpolation across novel views and time. Please visit our project page for visual results: https://ufo-4d.github.io/
Engineering Breakdown
Plain English
UFO-4D solves the problem of reconstructing dynamic 3D scenes (4D = 3D + time) from just two unposed images—meaning the camera position and orientation are unknown. Previous methods either require slow optimization at test time or use fragmented models designed for specific tasks. This paper presents a unified feedforward neural network that directly estimates Dynamic 3D Gaussian Splats, simultaneously solving for 3D geometry, 3D motion, and camera pose in a single forward pass. The key insight is that rendering multiple signals (appearance, depth, motion) from a shared geometric representation creates strong training signals and tight coupling between tasks, enabling self-supervised learning without ground truth labels.
Core Technical Contribution
The core innovation is a unified feedforward architecture that treats 4D reconstruction as a single end-to-end prediction problem rather than solving geometry, motion, and pose separately. Instead of optimizing parameters at test time like NeRF-based methods, UFO-4D predicts explicit Dynamic 3D Gaussian Splats directly—a parametric representation that is both efficient to render and easy to optimize. The authors discovered that differentiably rendering multiple modalities (RGB, depth, motion) from the same Gaussian representation creates strong self-supervised training signals; appearance and depth losses constrain the geometry while motion losses constrain the dynamics. This multi-modal rendering strategy from a unified representation is what eliminates the need for task-specific architectures and test-time optimization.
How It Works
The input is a pair of unposed RGB images from unknown camera viewpoints. The feedforward network processes these images through an encoder to extract features, then outputs 4D Gaussian parameters: position, covariance, spherical harmonics coefficients (for appearance), opacity, and per-frame velocity/motion vectors. These Gaussians are then differentiably rendered using splatting to produce: (1) RGB reconstruction for both input frames, (2) depth maps for geometric consistency, and (3) flow/motion estimates by rendering velocity. The network is trained end-to-end using multi-task losses: photometric loss on RGB, depth smoothness and consistency, and motion consistency across the Gaussian motion field. The shared Gaussian representation forces all three modalities to respect the same 3D geometry, creating a tightly coupled optimization landscape that reduces the solution space and enables learning from unposed image pairs alone.
Production Impact
For systems requiring dynamic 3D scene understanding (AR/VR content creation, autonomous driving 3D perception, video-based 3D reconstruction), UFO-4D eliminates two major bottlenecks: camera pose estimation and slow per-scene optimization. A production pipeline could capture two arbitrary images of a dynamic scene and immediately get dense 3D geometry, motion vectors, and camera poses without manual annotation or minutes of optimization. The feedforward design means inference is fast—a single forward pass replaces iterative optimization, enabling real-time or near-real-time reconstruction in applications like mobile AR or live video processing. Trade-offs include: the model requires training on large paired-image datasets (not shown in abstract but standard for feedforward methods), it may struggle with severely occluded or textureless regions where multiple signals can't constrain geometry, and the 3D Gaussian representation may be less compact than traditional mesh-based outputs for downstream storage or transmission.
Limitations and When Not to Use This
The paper does not address highly dynamic scenes with occlusions, large motion between frames, or significant illumination changes—conditions where the shared geometric representation may fail to disambiguate between motion and appearance variation. It assumes the two input images are temporally close enough that motion is smooth and small, limiting applicability to wide-baseline stereo or large temporal gaps. The method's dependence on self-supervised losses (no ground truth 3D or camera pose needed) may plateau in performance compared to supervised methods; the abstract doesn't report quantitative accuracy metrics or failure case analysis. Additionally, the Gaussian Splatting representation, while fast, may not generalize well to novel views or scales beyond the training distribution, and there's no discussion of how the method handles scenes with transparent or reflective surfaces where rendering multiple modalities becomes ambiguous.
Research Context
UFO-4D builds on the recent success of 3D Gaussian Splatting as an efficient 3D representation, extending it to the dynamic (4D) setting and adding the constraint of pose estimation from unposed images. It competes with and improves upon prior work in dense reconstruction from unposed images (which typically use NeRF-based test-time optimization like COLMAP or SfM pipelines) and dynamic 3D reconstruction from video. The work is part of a broader shift in computer vision toward feedforward, end-to-end learnable approaches that replace classical geometric pipelines—similar to how PoseNet replaced SIFT+RANSAC for pose estimation. This research opens directions for multi-image reconstruction, handling longer video sequences, and extending the approach to handle larger pose ambiguities and more complex dynamics in real-world scenarios.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::