ReconPhys: Reconstruct Appearance and Physical Attributes from Single Video
| Authors | Boyuan Wang et al. |
| Year | 2026 |
| HF Upvotes | 9 |
| arXiv | 2604.07882 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Reconstructing non-rigid objects with physical plausibility remains a significant challenge. Existing approaches leverage differentiable rendering for per-scene optimization, recovering geometry and dynamics but requiring expensive tuning or manual annotation, which limits practicality and generalizability. To address this, we propose ReconPhys, the first feedforward framework that jointly learns physical attribute estimation and 3D Gaussian Splatting reconstruction from a single monocular video. Our method employs a dual-branch architecture trained via a self-supervised strategy, eliminating the need for ground-truth physics labels. Given a video sequence, ReconPhys simultaneously infers geometry, appearance, and physical attributes. Experiments on a large-scale synthetic dataset demonstrate superior performance: our method achieves 21.64 PSNR in future prediction compared to 13.27 by state-of-the-art optimization baselines, while reducing Chamfer Distance from 0.349 to 0.004. Crucially, ReconPhys enables fast inference (<1 second) versus hours required by existing methods, facilitating rapid generation of simulation-ready assets for robotics and graphics.
Engineering Breakdown
Plain English
ReconPhys is a feedforward neural network that reconstructs 3D geometry, appearance, and physical properties (like material density, friction, elasticity) from a single monocular video without requiring ground-truth physics labels or manual annotation. The key innovation is eliminating the expensive per-scene optimization that prior methods required — instead, the model trains once on a large synthetic dataset and then generalizes to new videos at inference time. The method uses a dual-branch architecture where one branch estimates physical attributes and another performs 3D Gaussian Splatting reconstruction, both trained via self-supervised learning. This is the first feedforward approach to jointly solve appearance reconstruction and physics estimation from video, making it significantly more practical than optimization-based baselines that required expensive tuning for each new object.
Core Technical Contribution
The core novelty is combining physical attribute estimation with 3D Gaussian Splatting in a single feedforward network trained self-supervised, eliminating the need for ground-truth physics labels or per-scene optimization. Prior work either focused on geometry/appearance alone (using differentiable rendering with per-scene optimization) or required explicit physics annotations, both limiting scalability and generalizability. The dual-branch architecture is designed so that physics predictions and geometric reconstruction inform each other during training — the physics branch learns material properties while the 3D-GS branch learns viewpoint-consistent geometry and appearance. The self-supervised training strategy is critical: it allows the network to learn physically plausible reconstructions from video alone, without requiring labeled datasets of object mass, friction coefficients, or other physical parameters.
How It Works
The pipeline takes a monocular video sequence as input and processes it through a CNN-based feature extractor that builds a temporal representation of the object's motion and deformation. The dual-branch architecture splits here: the physics branch takes the video features and directly predicts dense physical attributes (scalar fields representing material properties per 3D location), while the geometry branch reconstructs the scene using 3D Gaussian Splatting, which represents the scene as a set of 3D Gaussians with learnable mean, covariance, color, and opacity. The two branches are coupled through a consistency loss during training — the predicted physics should allow the observed motion to be physically plausible when simulated forward. During inference, the network runs once on the video to produce both a 3D-GS reconstruction (which can be rendered from any viewpoint) and dense physical attribute maps, enabling downstream physics simulation or further analysis. The self-supervised loss likely includes photometric consistency (rendered output matches input frames) and temporal consistency (motion across frames respects physical plausibility constraints).
Production Impact
For teams building systems that reconstruct 3D objects from video, this eliminates the expensive optimization loop that currently dominates production pipelines — instead of running hours of gradient descent per video, you run the feedforward network once, dramatically reducing latency from hours to seconds. If your application needs to preserve physical properties (e.g., simulating cloth dynamics, predicting how objects deform under load, enabling physics-aware VFX), ReconPhys gives you physics predictions automatically without manual annotation of material properties. Integration is straightforward: the method outputs standard 3D-GS representations and scalar physical fields, both compatible with existing rendering engines and physics simulators. The main trade-off is dataset dependency — the model trains on synthetic data, so there is likely a sim-to-real gap you need to validate; you would need to fine-tune or validate on real footage of your specific object categories. Compute cost shifts from per-scene inference-time optimization to upfront training cost, making this more suitable for applications processing many videos rather than one-off reconstructions.
Limitations and When Not to Use This
The method trains on synthetic data, so real-world generalization remains unvalidated in the abstract — objects with non-rigid dynamics, complex materials, or lighting conditions outside the training distribution may fail. The approach assumes monocular video is sufficient for reconstruction, which inherently struggles with occlusions and thin structures that would benefit from multi-view input. Physics accuracy is not directly validated — while the network learns physically plausible reconstructions through self-supervision, there is no evidence it recovers ground-truth material properties or accurately simulates subsequent dynamics. The method likely requires relatively stable video with clear motion signal; fast motions, severe motion blur, or static scenes may produce poor physics estimates since the self-supervised signal depends on inferring physics from observable deformation.
Research Context
This work builds on the recent success of 3D Gaussian Splatting as a fast alternative to NeRF-based reconstruction, extending it from pure geometry/appearance to include physical reasoning. It addresses a known limitation in differentiable rendering approaches: per-scene optimization is slow and doesn't generalize, so a feedforward model trained on large synthetic data is a natural next step. The paper sits at the intersection of 3D reconstruction (computer vision) and physics-informed ML, opening a direction where neural networks learn to recover latent physical properties from video — this is relevant for downstream simulation, AR/VR content creation, and robotics applications that need physical understanding. The self-supervised training strategy is particularly noteworthy because it sidesteps the costly annotation problem; future work likely extends this to multi-view video, dynamic scene understanding, and real-to-sim adaptation for robotics.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::