CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning
| Authors | Ankan Deria et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.03231 |
| Download | |
| Categories | cs.CV |
Abstract
Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.
Engineering Breakdown
Plain English
CoME-VL proposes a modular framework that combines two complementary vision encoders for vision-language models: a contrastively trained encoder (CLIP-style) and a self-supervised DINO encoder. The paper addresses a real limitation in current VLMs—that single encoders trained with contrastive objectives excel at cross-modal alignment but miss dense semantic information that self-supervised methods capture better. The approach uses entropy-guided multi-layer aggregation and orthogonal fusion to integrate these representations, aiming to scale vision-language learning by leveraging the strengths of both encoder types simultaneously.
Core Technical Contribution
The core innovation is a dual-encoder fusion strategy that recognizes contrastive and self-supervised vision encoders capture fundamentally different information, then proposes a principled way to combine them at the representation level rather than concatenating or simply averaging. The key technical components are entropy-guided aggregation (selectively combining features from different layers based on information content) and orthogonal fusion constraints (ensuring the fused representations maintain diversity and don't collapse to redundancy). This is novel because most VLMs use a single encoder; recognizing and operationalizing encoder complementarity is a fresh direction that challenges the "bigger single encoder" paradigm.
How It Works
The system takes an image input and processes it through two parallel vision encoders: a CLIP-like contrastive encoder trained on image-text pairs, and a DINO encoder trained with self-supervised learning on images alone. Each encoder produces multi-layer intermediate representations (not just final embeddings). The entropy-guided aggregation module computes information entropy across layers and selectively combines high-information features from each encoder at multiple levels. The orthogonal fusion constraint ensures fused representations remain linearly independent using techniques like orthogonal projections, preventing feature collapse. The final fused representation feeds into a text encoder and language model head for vision-language tasks (captioning, VQA, retrieval). The modular design means contrastive and self-supervised components train with their standard objectives, then fusion happens during inference or fine-tuning.
Production Impact
Adopting this approach would mean running two vision encoders instead of one, roughly doubling vision encoder compute and memory during inference—a significant cost for latency-critical applications like real-time video understanding. However, you'd gain robustness on downstream recognition tasks and denser semantic understanding, reducing the need for task-specific fine-tuning in some scenarios. In a production pipeline, this works well for batch processing (documents, images, logs) where 2-3x latency increase is acceptable, but for mobile or edge deployment, you'd need quantization or distillation strategies. The modular design is a win: you can swap encoders independently, reuse pretrained weights from both CLIP and DINO ecosystems, and gradually adopt the approach without retraining everything.
Limitations and When Not to Use This
The paper doesn't clearly address the compute and memory overhead of dual encoders at scale—running CLIP and DINO in parallel during inference is expensive, and the paper may not discuss efficient deployment options like distillation or pruning. Orthogonal fusion constraints add complexity and hyperparameter tuning; it's unclear how sensitive the approach is to fusion weight initialization or how it generalizes to very different domain shifts (medical imaging, satellite data). The abstract is incomplete (cuts off mid-sentence), so critical details on training procedures, convergence behavior, and failure modes are missing. The paper likely assumes both encoders are available and well-pretrained; it doesn't address cold-start scenarios or how to handle encoder mismatch in continual learning settings.
Research Context
This builds directly on CLIP-style contrastive VLM pretraining and DINO self-supervised learning, asking the natural follow-up question: why choose one when both capture different signals? The work aligns with recent trends in multi-modal learning that move beyond single-encoder bottlenecks (similar motivation to mixture-of-experts and ensemble approaches). It likely benchmarks on standard VLM evaluations (ImageNet classification, COCO captioning, Flickr30K retrieval) and may introduce comparisons showing complementarity empirically. The research direction opens up investigation of other encoder combinations (BYOL, SimSiam, masked autoencoders) and principled fusion strategies beyond entropy weighting.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::