SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion
| Authors | Zhaoyang Li et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2605.01466 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Although multi-modal learning has advanced point cloud completion, the theoretical mechanisms remain unclear. Recent works attribute success to the connection between modalities, yet we identify that standard hard projection severs this connection: projecting a sparse point cloud onto the image plane yields an extremely sparse support, which hinders visual prior propagation, a failure mode we term Cross-Modal Entropy Collapse. To address this practical limitation, we propose SplAttN, which replaces hard projection with Differentiable Gaussian Splatting to produce a dense, continuous image-plane representation. By reformulating projection as continuous density estimation, SplAttN avoids collapsed sparse support, facilitates gradient flow, and improves cross-modal connection learnability. Extensive experiments show that SplAttN achieves state-of-the-art performance on PCN and ShapeNet-55/34. Crucially, we utilize the real-world KITTI benchmark as a stress test for multi-modal reliance. Counter-factual evaluation reveals that while baselines degenerate into unimodal template retrievers insensitive to visual removal, SplAttN maintains a robust dependency on visual cues, validating that our method establishes an effective cross-modal connection. Code is available at https://github.com/zay002/SplAttN.
Engineering Breakdown
Plain English
This paper identifies a fundamental failure mode in multi-modal point cloud completion called Cross-Modal Entropy Collapse, where projecting sparse 3D point clouds onto 2D image planes creates extremely sparse representations that break the connection between 3D and visual information. The authors propose SplAttN, which replaces traditional hard projection with Differentiable Gaussian Splatting to create dense, continuous 2D representations that preserve cross-modal information flow. This approach enables better gradient propagation and improves how visual priors from images can be used to complete missing parts of point clouds, addressing a practical limitation that prior multi-modal methods overlooked.
Core Technical Contribution
The core novelty is reformulating the projection operation in multi-modal point cloud completion as a continuous density estimation problem rather than a discrete hard projection. Instead of mapping sparse 3D points to discrete 2D pixel locations (causing information loss), SplAttN uses Differentiable Gaussian Splatting to create a smooth, dense probability distribution on the image plane. This maintains gradient connectivity between the point cloud and image domains, allowing the network to learn better cross-modal correspondences. The key insight is that the sparsity of hard projection actively prevents learning, and treating projection as density estimation solves this fundamental problem.
How It Works
The input to SplAttN is a partial point cloud (incomplete 3D data) and corresponding RGB images. Instead of projecting each sparse 3D point to a discrete 2D location, the method uses Differentiable Gaussian Splatting: each 3D point is converted into a Gaussian kernel on the 2D image plane, creating overlapping continuous representations. These Gaussians blend together to form a dense feature map that preserves spatial relationships while remaining fully differentiable. The dense 2D representation is then fused with image-based visual features, allowing the network to propagate visual priors (colors, textures, edges from images) back to guide point cloud completion. Gradients can flow smoothly through both the projection operation and the fusion mechanism, enabling the network to learn which visual information is relevant for predicting missing 3D geometry.
Production Impact
In production 3D reconstruction systems (autonomous vehicles, robotics, AR/VR), adopting SplAttN would improve completion quality when working with sparse LiDAR or depth sensor data combined with camera images. The practical benefit is better handling of occluded regions and noisy sensor data without requiring manual alignment between modalities. Trade-offs include: (1) increased computational cost from Gaussian splatting and dense feature maps compared to sparse projection, likely 2-3x slower than hard projection baselines, (2) memory overhead for maintaining dense 2D representations during inference, (3) requirement for well-calibrated camera-to-3D sensor extrinsics, and (4) integration complexity since it changes how projection layers are implemented in existing multi-modal architectures. For applications requiring real-time inference (robotics), this cost may be prohibitive; for offline reconstruction pipelines (3D scanning, VFX), the quality gains justify the compute.
Limitations and When Not to Use This
The paper assumes paired point cloud and image data with known camera calibration, which may not hold in practice with poorly synchronized or uncalibrated sensors. The approach adds computational overhead through Gaussian splatting, making it impractical for extremely latency-sensitive applications; the paper does not provide detailed runtime analysis or show how to optimize for edge deployment. The method is only demonstrated on point cloud completion tasks; it's unclear if the Cross-Modal Entropy Collapse problem and the Gaussian splatting solution generalize to other multi-modal architectures or 3D tasks like segmentation or detection. Additionally, the approach still requires dense image features, limiting applicability in scenarios with low-resolution or degraded image inputs.
Research Context
This work builds on recent advances in multi-modal 3D understanding and Gaussian splatting techniques (which have been successful in novel view synthesis). It directly addresses a gap in prior multi-modal point cloud completion methods (like ULIP, CLiP-3D variants) that didn't analyze the information loss caused by hard projection. The paper contributes to a growing understanding of how 3D and 2D representations should interact in modern vision systems, following the trend of replacing discrete operations with continuous, differentiable alternatives. Future research likely builds on this to apply differentiable splatting to other 3D tasks and to develop more efficient variants that maintain the gradient-flow benefits without prohibitive compute costs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::