DENALI: A Dataset Enabling Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs
| Authors | Nikhil Behari et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2604.16201 |
| Download | |
| Categories | cs.RO, cs.CV |
Abstract
Consumer LiDARs in mobile devices and robots typically output a single depth value per pixel. Yet internally, they record full time-resolved histograms containing direct and multi-bounce light returns; these multi-bounce returns encode rich non-line-of-sight (NLOS) cues that can enable perception of hidden objects in a scene. However, severe hardware limitations of consumer LiDARs make NLOS reconstruction with conventional methods difficult. In this work, we motivate a complementary direction: enabling NLOS perception with low-cost LiDARs through data-driven inference. We present DENALI, the first large-scale real-world dataset of space-time histograms from low-cost LiDARs capturing hidden objects. We capture time-resolved LiDAR histograms for 72,000 hidden-object scenes across diverse object shapes, positions, lighting conditions, and spatial resolutions. Using our dataset, we show that consumer LiDARs can enable accurate, data-driven NLOS perception. We further identify key scene and modeling factors that limit performance, as well as simulation-fidelity gaps that hinder current sim-to-real transfer, motivating future work toward scalable NLOS vision with consumer LiDARs.
Engineering Breakdown
Plain English
This paper introduces DENALI, the first large-scale real-world dataset of time-resolved LiDAR histograms capturing hidden objects using consumer-grade LiDARs. Rather than using conventional physics-based methods for non-line-of-sight (NLOS) reconstruction—which struggle with the noisy, limited hardware in cheap mobile and robotic LiDARs—the authors propose a data-driven approach that learns to perceive occluded objects directly from histogram data. The dataset contains 72,000 scenes of hidden objects, enabling training of deep learning models to extract NLOS perception from the raw multi-bounce light returns that consumer LiDARs already record internally but typically discard. This work opens a practical path to enabling perception-through-walls and around-corner detection on mass-market devices.
Core Technical Contribution
The core novelty is reframing NLOS perception as a supervised learning problem rather than an inverse rendering problem. Most prior NLOS work relies on time-of-flight physics simulation and expensive computational reconstruction; this paper instead treats the full time-resolved histogram from a consumer LiDAR as a neural network input and directly regresses or classifies hidden object properties. The key insight is that consumer LiDARs already capture multi-bounce photons in their internal histogram buffers—they just throw this data away at the hardware output stage. By collecting the first large-scale real-world dataset of these histograms paired with ground truth hidden object locations, the authors enable end-to-end learning without requiring accurate hardware calibration, ray-tracing simulation, or iterative reconstruction algorithms.
How It Works
The technical pipeline begins with consumer LiDARs (e.g., those in iPhones or mobile robots) capturing time-resolved histograms: for each pixel, a histogram bin records how many photons arrived at each time delay, encoding both direct (line-of-sight) returns and multi-bounce (NLOS) returns. The raw histogram is the input—typically a tensor of shape [height, width, time_bins], containing raw photon counts across nanosecond-scale time windows. A neural network (likely a CNN or 3D convolutional architecture, though specifics aren't in the abstract) processes this histogram volume to predict properties of hidden objects: their location, size, or class. The network learns to correlate patterns in late-arriving photons (which traveled longer paths through the scene) with the presence and position of occluded objects. The training is supervised using ground truth labels: during data collection, the authors manually annotated or measured the hidden object locations for each of the 72,000 scenes. At inference time, a mobile device or robot can run this learned model in real-time on raw histogram data without requiring physics simulation or iterative optimization.
Production Impact
For engineers building mobile robotics or augmented reality systems, this enables a genuinely new capability: detecting people, obstacles, or objects around corners or behind walls using only passive LiDAR data already present in the hardware. In a production pipeline, you would intercept the raw histogram buffer from the LiDAR IC before it is processed down to a single depth value, feed it through a small neural network (likely quantizable to 4-8 bit integer arithmetic for mobile inference), and get NLOS object detections with a few milliseconds of latency. This is far cheaper and more practical than adding a second sensor or custom hardware. The trade-off is that you need the DENALI dataset or similar real-world training data; synthetic or simulated histograms do not generalize well due to the complexity of real photon noise and multi-path effects. Memory footprint is modest (histograms are small tensors), but you do require access to raw histogram data—some LiDAR vendors lock this behind proprietary firmware, so integration depends on hardware partnerships.
Limitations and When Not to Use This
The paper does not address generalization across different LiDAR hardware; the dataset likely comes from one or a few specific consumer LiDAR models, so models trained on it may not transfer to different hardware with different histogram resolution, timing, or noise characteristics. Outdoor scenes and strong ambient light are likely underrepresented, since most consumer LiDARs struggle in bright sunlight—NLOS reconstruction in outdoor settings remains unsolved. The method assumes relatively simple hidden object geometry and scenes; reconstructing detailed shapes or multiple occluded objects in complex clutter is not addressed. Finally, there is a fundamental physical limit: if an object is hidden beyond multiple bounces, or if the scene scatters photons diffusely, the NLOS signal vanishes into noise—the data-driven approach cannot overcome this, only optimize for the cases where photons do return.
Research Context
This work builds on decades of computational photography research in NLOS imaging (e.g., prior work using time-of-flight cameras and expensive laboratory setups), but pivots to the practical constraint of real consumer hardware. It parallels recent trends in learned computational imaging: using neural networks to extract signal from raw sensor histograms rather than relying on traditional inverse problems. The DENALI dataset itself is a major research contribution, similar to how large-scale vision datasets (COCO, ImageNet) enabled deep learning in computer vision; it will likely become a benchmark for NLOS perception research. The work opens a new direction in implicit neural sensing—the idea that cheap, noisy sensors combined with learned models can solve problems previously thought to require expensive, specialized hardware.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::