A Mixed Diet Makes DINO An Omnivorous Vision Encoder
| Authors | Rishabh Kabra et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24181 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their feature representations are poorly aligned across different modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a novel framework that learns a modality-agnostic feature space. We train the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to the output of a fully frozen teacher such as DINOv2. The resulting student encoder becomes "omnivorous" by producing a consistent, powerful embedding for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model.
Engineering Breakdown
Plain English
This paper addresses a fundamental limitation of DINOv2 and similar pre-trained vision encoders: they learn representations that work well for single-modality tasks (like RGB images alone) but fail to align features across different modalities of the same scene. The authors show that an RGB image and its corresponding depth map produce embeddings with nearly zero meaningful correlation—essentially as random as completely unrelated images. They propose the Omnivorous Vision Encoder, which uses a dual training objective combining cross-modal alignment loss with knowledge distillation from a frozen DINOv2 teacher to learn a unified feature space that works across RGB, depth, and potentially other modalities while preserving the strong unimodal performance of the original encoder.
Core Technical Contribution
The core innovation is a training framework that simultaneously optimizes two objectives: maximizing cosine similarity between embeddings of different modalities depicting the same scene, while maintaining fidelity to a frozen teacher encoder via distillation. This allows the model to learn modality-agnostic representations without catastrophic forgetting of the strong unimodal features DINOv2 already learned. The key insight is that you don't need to retrain from scratch—instead, you can anchor the new cross-modal learning to a frozen teacher, preventing the model from degrading on tasks where the original encoder excels. This approach elegantly sidesteps the typical trade-off between multi-modal robustness and single-modal performance by treating distillation as a regularization mechanism that preserves the original feature quality.
How It Works
The encoder takes as input an image in any modality (RGB, depth, infrared, etc.) and produces a feature embedding through the standard backbone architecture inherited from DINOv2. During training, the model processes paired images of the same scene in different modalities and computes a cross-modal alignment loss that pulls their embeddings closer in feature space (likely using contrastive or similarity-based losses). Simultaneously, the model computes a distillation loss by comparing its output embeddings against the frozen teacher DINOv2 embeddings, ensuring that the new multi-modal encoder doesn't drift too far from the original solution. The weighted combination of these two losses guides the encoder to learn a shared representation space where depth, RGB, thermal, and other modalities naturally cluster together for the same scene, while the distillation term keeps the learned features sufficiently close to the teacher's solution that single-modality performance is preserved or even improved.
Production Impact
For engineers building multi-modal perception systems (autonomous driving, robotics, 3D scene understanding), this approach eliminates the need to maintain separate feature extractors for each modality or retrain a full vision model from scratch. In production, you could deploy a single omnivorous encoder that handles RGB, depth, and other sensors interchangeably, reducing model serving complexity and memory footprint. The distillation-based training means you inherit DINOv2's already-excellent single-modality performance as a baseline, so you don't risk regression on existing single-modal downstream tasks when deploying the new encoder. However, the training requires paired multi-modal data (e.g., RGB-depth pairs from the same scene), which may require additional data collection or synthetic alignment if your existing datasets are unimodal. The computational cost during training increases due to processing multiple modalities and computing two loss objectives, though inference latency should be identical to DINOv2 since the architecture doesn't change.
Limitations and When Not to Use This
The paper assumes you have access to well-aligned multi-modal data during training (RGB-depth pairs, for example), which is a significant constraint if your domain lacks such paired datasets or if modality alignment is noisy. The approach's effectiveness likely depends heavily on the quality of the teacher model and the weight balance between the distillation and alignment losses—there's a risk of under-explored hyperparameter sensitivity in production settings. The paper doesn't clearly address how the method scales to more than two modalities or highly disparate modalities (e.g., can it align radar with RGB as effectively as depth with RGB?), leaving questions about generalization. Additionally, the frozen teacher approach means you're somewhat constrained to the original DINOv2 solution space; if a completely different feature geometry would be optimal for multi-modal tasks, the distillation constraint may prevent the model from discovering it.
Research Context
This work builds directly on DINOv2, which demonstrated that self-supervised learning on unimodal vision data produces remarkably strong general-purpose feature encoders. The paper extends the vision encoder paradigm from single-modality to multi-modality, a natural evolution as real-world systems increasingly deploy multi-sensor stacks. It relates to broader research on cross-modal learning and modality alignment (e.g., CLIP for vision-language), but focuses on aligning similar visual modalities rather than semantically different modalities. The distillation-based approach connects to recent work showing that frozen teachers can serve as effective regularizers for learning new capabilities without catastrophic forgetting, opening a research direction toward building increasingly capable encoders by layering new training objectives atop strong frozen baselines rather than retraining from scratch.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::