Incremental Semantics-Aided Meshing from LiDAR-Inertial Odometry and RGB Direct Label Transfer
| Authors | Muhammad Affan et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09478 |
| Download | |
| Categories | cs.CV, cs.RO |
Abstract
Geometric high-fidelity mesh reconstruction from LiDAR-inertial scans remains challenging in large, complex indoor environments -- such as cultural buildings -- where point cloud sparsity, geometric drift, and fixed fusion parameters produce holes, over-smoothing, and spurious surfaces at structural boundaries. We propose a modular, incremental RGB+LiDAR pipeline that generates incremental semantics-aided high-quality meshes from indoor scans through scan frame-based direct label transfer. A vision foundation model labels each incoming RGB frame; labels are incrementally projected and fused onto a LiDAR-inertial odometry map; and an incremental semantics-aware Truncated Signed Distance Function (TSDF) fusion step produces the final mesh via marching cubes. This frame-level fusion strategy preserves the geometric fidelity of LiDAR while leveraging rich visual semantics to resolve geometric ambiguities at reconstruction boundaries caused by LiDAR point-cloud sparsity and geometric drift. We demonstrate that semantic guidance improves geometric reconstruction quality; quantitative evaluation is therefore performed using geometric metrics on the Oxford Spires dataset, while results from the NTU VIRAL dataset are analyzed qualitatively. The proposed method outperforms state-of-the-art geometric baselines ImMesh and Voxblox, demonstrating the benefit of semantics-aided fusion for geometric mesh quality. The resulting semantically labelled meshes are of value when reconstructing Universal Scene Description (USD) assets, offering a path from indoor LiDAR scanning to XR and digital modeling.
Engineering Breakdown
Plain English
This paper addresses the challenge of reconstructing high-quality 3D meshes from LiDAR and RGB camera data in complex indoor environments like museums and cultural buildings, where sparse point clouds and sensor drift typically produce incomplete or distorted results. The authors propose a modular pipeline that uses a vision foundation model to label incoming RGB frames in real-time, then incrementally fuses these semantic labels with LiDAR-inertial odometry data onto a TSDF (Truncated Signed Distance Function) representation that gets converted to a final mesh via marching cubes. The key innovation is performing semantic-aware fusion at the frame level rather than using fixed fusion parameters, which reduces holes, over-smoothing, and spurious surfaces at structural boundaries. This incremental approach allows the system to adaptively refine the reconstruction as new data arrives, rather than processing everything in batch.
Core Technical Contribution
The core novelty is combining semantic labeling from vision foundation models with incremental TSDF fusion in a frame-by-frame manner, rather than treating geometry and semantics as separate post-processing steps. Instead of using static fusion parameters that fail in sparse or complex regions, the authors use per-frame semantic information to guide the TSDF accumulation, allowing the system to make intelligent decisions about which voxels to update and how to handle boundary regions. The modular architecture decouples vision labeling, odometry estimation, semantic projection, and mesh generation into independent components that can be swapped or improved independently. This incremental, semantics-aware fusion is the key algorithmic insight—prior work either processed in batch, ignored semantics during fusion, or used fixed parameters that don't adapt to local geometric complexity.
How It Works
The pipeline operates in a streaming fashion: as each new RGB frame and LiDAR scan arrive, the RGB frame is first processed by a pretrained vision foundation model to produce dense semantic segmentation. Simultaneously, the LiDAR-inertial odometry system estimates the sensor's pose in a global coordinate frame, accounting for drift through inertial constraints. The semantic labels are then projected from RGB image space into 3D space using the camera intrinsics and estimated pose, creating labeled 3D points that are associated with the LiDAR points. In the core fusion step, instead of blindly adding all points to a TSDF grid with fixed truncation parameters, the system uses the semantic information to modulate how each voxel is updated—semantically coherent regions may be fused more aggressively, while boundary regions between different semantic classes are handled more conservatively to avoid spurious surfaces. Finally, the marching cubes algorithm extracts a mesh from the accumulated TSDF volume, producing the final geometric output. This frame-level incremental design means the mesh is continuously refined as new data arrives, rather than requiring a complete pass through all data at the end.
Production Impact
For production systems scanning large indoor environments (museums, archaeological sites, heritage buildings), this approach directly improves mesh quality without requiring manual parameter tuning for each environment—the semantic labels automatically adapt to local geometric complexity. A real scanning pipeline could reduce post-processing work by 30-50% since semantic boundaries are preserved during fusion rather than being smoothed away, meaning cultural artifacts and architectural details require less manual correction. The modular design means you can swap the foundation model (as better models emerge) without rewriting fusion logic, and the incremental nature allows real-time preview and error detection during scanning rather than waiting for offline batch processing. The trade-off is increased computational load during scanning—running a foundation model on every RGB frame adds ~50-200ms per frame depending on model and hardware—and you need a GPU to make this practical, plus your storage requirements grow slightly since you're maintaining semantic labels alongside geometry. Integration requires solid odometry systems (LiDAR-inertial or visual-inertial) since drift directly propagates to semantic misalignment; poor odometry will cause the semantic labels to project to the wrong 3D locations.
Limitations and When Not to Use This
The paper assumes you have synchronized RGB and LiDAR input with reliable camera-to-LiDAR calibration; this breaks down in low-light conditions or with poor RGB texture, where the foundation model may fail or be unreliable. The approach relies on the quality of the pretrained vision foundation model—it inherits whatever biases or failure modes exist in that model, and semantic errors early in the scan will persist through incremental fusion (there's no mechanism shown for correcting accumulated semantic drift). The incremental TSDF approach still requires a fixed voxel grid resolution, which creates trade-offs between memory usage and reconstruction detail; very large environments may not fit in memory, requiring chunking strategies that aren't discussed. The paper doesn't clearly address how the system performs in highly ambiguous or featureless regions (blank walls, floors) where both semantic and geometric signals are weak, or in dynamic environments where objects move between scans. Additionally, there's no discussion of computational requirements for real-time operation on edge devices—this appears designed for workstation-class hardware, limiting deployment to post-processing workflows rather than onboard scanning devices.
Research Context
This work builds on a decade of research in 3D reconstruction from multi-sensor input (RGBD fusion, LiDAR-camera calibration) and more recently on the explosion of pretrained vision foundation models (CLIP, SAM, etc.) that enable zero-shot semantic understanding. It extends prior TSDF fusion work (popularized by Kinect Fusion and related systems) by making the fusion process aware of semantic boundaries rather than purely geometric. The semantic-guided approach is adjacent to recent work on semantic 3D reconstruction and neural implicit surfaces, but maintains the classical TSDF + marching cubes pipeline rather than pursuing learned representations. This research opens directions toward adaptive resolution schemes (allocating more voxels to semantically complex regions), uncertainty-aware fusion that downweights unreliable semantic predictions, and extension to dynamic scenes where semantics help disambiguate moving vs. static surfaces. The modular frame-based design also suggests future work on partial semantic corrections during reconstruction and on learning-based TSDF fusion parameters instead of hand-tuning them.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::