C-GenReg: Training-Free 3D Point Cloud Registration by Multi-View-Consistent Geometry-to-Image Generation with Probabilistic Modalities Fusion
| Authors | Yuval Haitman et al. |
| Year | 2026 |
| HF Upvotes | 15 |
| arXiv | 2604.16680 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We introduce C-GenReg, a training-free framework for 3D point cloud registration that leverages the complementary strengths of world-scale generative priors and registration-oriented Vision Foundation Models (VFMs). Current learning-based 3D point cloud registration methods struggle to generalize across sensing modalities, sampling differences, and environments. Hence, C-GenReg augments the geometric point cloud registration branch by transferring the matching problem into an auxiliary image domain, where VFMs excel, using a World Foundation Model to synthesize multi-view-consistent RGB representations from the input geometry. This generative transfer, preserves spatial coherence across source and target views without any fine-tuning. From these generated views, a VFM pretrained for finding dense correspondences extracts matches. The resulting pixel correspondences are lifted back to 3D via the original depth maps. To further enhance robustness, we introduce a "Match-then-Fuse" probabilistic cold-fusion scheme that combines two independent correspondence posteriors, that of the generated-RGB branch with that of the raw geometric branch. This principled fusion preserves each modality inductive bias and provides calibrated confidence without any additional learning. C-GenReg is zero-shot and plug-and-play: all modules are pretrained and operate without fine-tuning. Extensive experiments on indoor (3DMatch, ScanNet) and outdoor (Waymo) benchmarks demonstrate strong zero-shot performance and superior cross-domain generalization. For the first time, we demonstrate a generative registration framework that operates successfully on real outdoor LiDAR data, where no imagery data is available.
Engineering Breakdown
Plain English
C-GenReg is a training-free framework that solves 3D point cloud registration—the problem of aligning two separate point clouds in 3D space—by converting the geometric matching problem into the image domain where modern vision foundation models are more capable. Instead of relying on traditional learning-based methods that struggle when data comes from different sensors or environments, the paper leverages a World Foundation Model to generate consistent RGB images from the input point clouds, then uses a Vision Foundation Model pretrained for correspondence matching to find alignments. The key insight is that foundation models trained on massive internet-scale data have learned rich geometric understanding that transfers well across modalities and environments without any task-specific fine-tuning, making the approach more robust and generalizable than existing methods.
Core Technical Contribution
The core novelty is a modality-agnostic registration pipeline that bypasses the generalization limitations of learning-based point cloud networks by reformulating the problem in an auxiliary image space. Rather than training a custom registration network, C-GenReg orchestrates two off-the-shelf foundation models: a generative model that synthesizes geometrically-consistent multi-view RGB representations from point clouds, and a vision foundation model that performs correspondence finding on those generated views. This cross-domain transfer is training-free—requiring no fine-tuning or adapter layers—which allows the method to inherit the generalization properties of large-scale foundation models trained on diverse data. The technical contribution is showing that the inductive biases of image-domain VFMs (trained on billions of natural images) are sufficiently aligned with 3D geometric matching that they can outperform specialized 3D networks.
How It Works
The pipeline operates in three stages: (1) Input stage takes two unaligned point clouds from arbitrary sensors and environments; (2) Generative transfer stage feeds each point cloud to a World Foundation Model that synthesizes multiple consistent RGB views from different viewpoints, preserving the spatial coherence of the original geometry without distortion; (3) Matching stage passes the generated image pairs into a pretrained Vision Foundation Model (such as a DINO or similar dense correspondence model) that identifies pixel-level correspondences between the views. These image-domain correspondences are then back-projected into 3D space using camera intrinsics and depth information to recover point-to-point matches. Finally, a classical registration solver (e.g., RANSAC or SVD-based alignment) uses these correspondences to compute the rotation and translation that aligns source to target. The key mechanism is that the generative model produces geometrically accurate but photorealistic views that trigger the strong correspondence-finding capabilities of VFMs, while the VFM's learned features generalize across sensor types because they operate on natural image statistics rather than point-specific features.
Production Impact
This approach directly addresses a critical pain point in real-world robotics and autonomous systems: traditional point cloud registration networks trained on LiDAR fail when deployed on different sensors (e.g., structured light, RGB-D cameras) or in visually different environments. By making registration training-free and dependent only on foundation model inference, C-GenReg eliminates the need for expensive per-domain data collection and model retraining, dramatically reducing time-to-deployment for new hardware or environments. The inference pipeline is straightforward to integrate: replace your point cloud registration module with (1) calls to a generative model API, (2) a VFM correspondence pass, and (3) a lightweight 3D alignment routine, all of which are now commoditized cloud services or open-source weights. The primary trade-off is compute cost—synthesizing multiple consistent views and running VFM inference is slower than a specialized lightweight network, likely adding 200-500ms per registration compared to optimized learned methods, making it suitable for offline batch alignment or scenarios where robustness outweighs latency. Data requirements shift from needing paired point clouds for training to needing good foundation models, which are now widely available, reducing the practical engineering burden even though each inference is heavier.
Limitations and When Not to Use This
The paper assumes access to high-quality foundation models—if your generative model produces geometrically inconsistent views or your VFM is weak at correspondence, the whole pipeline fails gracefully. The approach requires known approximate camera intrinsics to back-project correspondences to 3D, which may not be trivial for novel sensor types without calibration. The method's generalization is bounded by the training data of the foundation models themselves; if your domain is drastically out-of-distribution (e.g., underwater point clouds with minimal natural image priors), the approach may not transfer as well as claimed. The paper likely only demonstrates results on relatively clean point clouds with sufficient geometric structure; sparse or extremely noisy clouds may produce unreliable views from the generative model. Follow-up work should explore (1) how to handle partial overlaps and occlusions, (2) end-to-end learning of the view synthesis component for better task alignment, and (3) quantitative benchmarks on diverse sensor types to validate the cross-modality claims.
Research Context
This work sits at the intersection of two major research directions: (1) the rise of world models and generative 3D priors that can synthesize geometrically coherent views from sparse input, and (2) the effectiveness of vision foundation models as universal correspondence engines that transfer across diverse visual tasks. It builds on the insight that foundation models—trained on massive unlabeled datasets—encode useful inductive biases that often outperform task-specific learning, a principle established in works like CLIP, DINO, and recent 3D foundation models. The paper improves upon traditional point cloud registration benchmarks (likely 3DMatch, 4PCS, or ModelNet) by showing better generalization across modalities without retraining. It opens research into hybrid approaches that leverage generative models not just for data augmentation or pretraining, but as core computational components in geometric pipelines, potentially inspiring similar reformulations in other 3D tasks like segmentation or completion.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::