Unlocking Dense Metric Depth Estimation in VLMs
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 3 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Hanxun Yu et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2605.15876 |
| Download | |
| Code | https://github.com/hanxunyu/DepthVLM |
Abstract
Vision-Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visual perception and prevents the recovery of dense geometry. Prior methods either distill geometry from external vision models, introducing error accumulation, or enable direct prediction with inefficient per-pixel query or coarse token-level outputs. In this paper, we propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor while preserving its multimodal capability. By attaching a lightweight depth head to the LLM backbone and training under a unified vision-text supervision paradigm with a two-stage schedule, DepthVLM generates full-resolution depth maps alongside language outputs in a single forward pass. We further introduce a unified indoor-outdoor metric depth benchmark in a VLM-compatible format. Experiments show that DepthVLM significantly outperforms existing VLMs with higher inference efficiency, surpasses leading pure vision models, and improves complex 3D spatial reasoning, moving toward a truly unified foundation model. All code and checkpoints will be publicly released.
Engineering Breakdown
Plain English
This paper introduces DepthVLM, a method to make Vision-Language Models (VLMs) predict dense 3D depth maps directly, rather than just handling 2D tasks like image captioning. The key innovation is attaching a lightweight depth prediction head to the LLM backbone and training it with both vision and text supervision in two stages, avoiding the error accumulation that comes from distilling geometry from separate models.
Key Engineering Insight
Instead of bolting on external vision models or running inefficient per-pixel queries, you can extend an existing VLM with a minimal depth head and train it end-to-end. This preserves the multimodal capabilities while adding dense 3D geometry understanding without architectural redesign.
Why It Matters for Engineers
VLMs are increasingly deployed in production for perception tasks, but they've been confined to 2D. If DepthVLM works reliably, you can now use a single model for both semantic understanding (from VLM) and geometric reasoning (depth) without running multiple models in parallel, reducing latency and inference cost in robotics, autonomous systems, and AR applications.
Research Context
Prior work tried either distilling depth from separate specialized models (introducing cascade errors) or adding inefficient per-pixel prediction heads. DepthVLM advances the field by showing that careful architecture and training strategy can make VLMs natively 3D-aware, enabling a more unified multimodal model that handles both language and geometry in one forward pass.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::