Spatial Calibration of Diffuse LiDARs
| Authors | Nikhil Behari & Ramesh Raskar |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06531 |
| Download | |
| Categories | cs.CV, cs.RO |
Abstract
Diffuse direct time-of-flight LiDARs report per-pixel depth histograms formed by aggregating photon returns over a wide instantaneous field of view, violating the single-ray assumption behind standard LiDAR-RGB calibration. We present a simple spatial calibration procedure that estimates, for each diffuse LiDAR pixel, its footprint (effective support region) and relative spatial sensitivity in a co-located RGB image plane. Using a scanned retroreflective patch with background subtraction, we recover per-pixel response maps that provide an explicit LiDAR-to-RGB correspondence for cross-modal alignment and fusion. We demonstrate the method on the ams OSRAM TMF8828.
Engineering Breakdown
Plain English
This paper solves a calibration problem that arises when using diffuse time-of-flight LiDARs (specifically the ams OSRAM TMF8828) alongside RGB cameras. Standard LiDAR-RGB calibration assumes each LiDAR pixel measures depth along a single ray, but diffuse LiDARs aggregate photon returns over a wide field of view per pixel, breaking this assumption. The authors present a spatial calibration method that maps out the exact footprint and sensitivity profile of each LiDAR pixel in the RGB image plane by scanning a retroreflective patch and subtracting background noise. This produces per-pixel response maps that enable accurate cross-modal alignment and fusion between LiDAR depth and RGB imagery.
Core Technical Contribution
The key novelty is the explicit characterization of diffuse LiDAR per-pixel response functions rather than treating each pixel as a point measurement. Instead of assuming a single ray per LiDAR pixel, the authors recover the spatial support region (footprint) and relative sensitivity distribution that describes how each diffuse LiDAR pixel maps to the RGB image plane. This is accomplished through a simple scanning procedure with retroreflective material and background subtraction—no complex optimization or learning required. This approach directly addresses a mismatch between how diffuse LiDARs physically behave and how they are conventionally calibrated, enabling reliable fusion of depth and color information.
How It Works
The calibration procedure works by physically scanning a retroreflective patch across the scene while simultaneously recording LiDAR and RGB data. For each position of the patch, the LiDAR sensor returns photon counts for each pixel, and the RGB camera captures the patch location. By subtracting background photon counts (ambient light, sensor noise) from the signal, the authors isolate the true response of each LiDAR pixel to the patch at different spatial positions. They repeat this scanning across the full field of view to build a per-pixel response map—essentially a 2D spatial sensitivity kernel for each LiDAR pixel that describes where it 'sees' in the RGB image plane and with what intensity. The output is a lookup table or dense correspondence map that allows any measurement from the diffuse LiDAR to be projected accurately to the RGB image and vice versa, enabling pixel-level fusion. This maps the discrete LiDAR pixel space (typically much coarser than RGB) to the continuous RGB space via empirical spatial kernels.
Production Impact
For engineers building autonomous systems, robotics, or depth-sensing applications using diffuse LiDARs, this work eliminates a major source of misalignment between depth and color modalities. In production pipelines, naive calibration of diffuse LiDARs leads to blur or offset in cross-modal fusion, degrading downstream tasks like object detection, segmentation, or 3D reconstruction. Adopting this calibration procedure requires a one-time offline phase where you scan your LiDAR-RGB rig with a retroreflective patch—this could take minutes to hours depending on desired spatial resolution and is similar in spirit to checkerboard calibration for standard camera pairs. Once calibrated, inference becomes a simple lookup of per-pixel response maps with minimal computational overhead. The trade-off is that calibration must be redone if cameras or LiDAR are physically repositioned or replaced, and the method assumes access to a retroreflective patch during setup, which is not always available in the field.
Limitations and When Not to Use This
This approach assumes stable ambient lighting and sensor behavior during calibration; if photon backgrounds drift significantly between calibration and deployment, accuracy degrades. The method also requires the LiDAR and RGB camera to be rigidly mounted and colocated—it does not address temporal synchronization issues or handle cases where sensors are mounted far apart. The paper focuses on a single sensor model (TMF8828) and does not validate generalization to other diffuse LiDAR designs, which may have different field-of-view characteristics or photon aggregation strategies. Additionally, the retroreflective patch scanning procedure assumes the patch is small enough to fit within individual LiDAR pixel footprints; for very large footprints or multi-path reflection scenarios, this assumption may break down. The work does not address what happens when LiDAR pixels see multiple objects at different depths simultaneously (common in real scenes), which could confound the per-pixel response map.
Research Context
This work builds on the long history of geometric calibration between depth and color sensors, tracing back to standard stereo and RGB-D calibration methods. Prior approaches treated LiDAR pixels as point measurements, applying projective geometry calibration similar to multi-camera rigs; this paper recognizes that diffuse time-of-flight LiDARs require a different model. The contribution is timely as diffuse direct time-of-flight sensing is becoming more common in consumer and automotive applications due to its compact form factor and robustness to ambient light. This opens research directions in per-sensor-element characterization for other emerging multi-modal sensors, generalization of calibration to non-Lambertian surfaces, and uncertainty quantification in diffuse LiDAR measurements. The work also connects to the broader trend of learning or characterizing sensor-specific response functions rather than applying one-size-fits-all calibration pipelines.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::