Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?
| Authors | Tilemachos Aravanis et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23339 |
| Download | |
| Categories | cs.CV |
Abstract
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision-language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and applies to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
Engineering Breakdown
Plain English
This paper tackles open-vocabulary segmentation (OVS) by combining vision-language models with few-shot learning to segment arbitrary object categories from text prompts. The core problem is that VLMs are trained on coarse image-level supervision, making them struggle with pixel-level predictions and semantic ambiguity in natural language descriptions. The authors propose augmenting text prompts with a small set of pixel-annotated support images, then use a retrieval-augmented test-time adapter that learns a lightweight per-image classifier by intelligently fusing textual and visual features. This learned fusion approach outperforms prior hand-crafted fusion methods and significantly closes the gap between zero-shot and fully supervised segmentation performance.
Core Technical Contribution
The key novelty is introducing a few-shot setting into open-vocabulary segmentation, where support images with pixel-level annotations bridge the supervision gap between VLM pretraining and dense prediction tasks. Rather than relying on late-stage, hand-crafted feature fusion strategies, the authors propose a learned fusion mechanism that adapts at test time on a per-image basis, leveraging both textual embeddings from language models and visual features from the support set. This retrieval-augmented approach enables the model to ground abstract language descriptions in concrete visual examples, reducing semantic ambiguity without requiring full dataset-level retraining. The test-time adaptation is lightweight and efficient, suggesting the adapter weights can be computed on-the-fly for each image using support features.
How It Works
The system operates in two stages: (1) Retrieval and preprocessing, where the support set of pixel-annotated images and the target image are encoded using a vision-language model backbone; textual prompts are also embedded into the same feature space. (2) Test-time adaptation, where a lightweight classifier is learned that fuses visual features from the support images with textual features from the prompt—crucially, this fusion is learned rather than hand-crafted, allowing the model to discover which modality and which support examples are most relevant for the current query. The per-image adapter takes as input the support features (visual embeddings with pixel-level masks indicating object locations) and query text embedding, and outputs a classifier that segments the target image. The learned fusion likely uses attention mechanisms or learned weighted combinations to determine how much to trust each support example and each modality, rather than fixed strategies like concatenation or simple averaging.
Production Impact
For production systems, this approach offers a practical way to extend segmentation capabilities to new, previously unseen object categories with minimal additional annotation burden—you need only a handful of labeled examples rather than thousands of full-image annotations. The test-time adaptation component is computationally lightweight, making it feasible to deploy on edge devices or in real-time applications, since no expensive retraining is required when encountering new categories or domains. However, the requirement for pixel-level annotations in support images is more expensive than image-level labels, so data collection strategy becomes critical; you'd need to balance the annotation cost against the performance gains from few-shot learning. Integration into existing pipelines requires a vector database for efficient retrieval of relevant support images and careful engineering around the per-image adaptation step to avoid latency bottlenecks during inference.
Limitations and When Not to Use This
The paper does not address how to effectively select or curate support sets—in production, having the wrong support examples could hurt performance, and the method's sensitivity to support set quality is likely unexplored. The approach still depends on the quality of the underlying VLM backbone; if the backbone has weak semantic understanding of certain domains or language variations, few-shot augmentation may not fully compensate. Scalability to very large numbers of categories or domains is unclear—it's unknown whether the per-image adapter can efficiently handle scenarios with hundreds of possible classes or whether retrieval becomes a bottleneck. The paper likely assumes support images come from the same distribution or domain as the test image; cross-domain generalization and robustness to distribution shift remain open questions.
Research Context
This work builds directly on recent progress in vision-language models (like CLIP) and zero-shot segmentation methods, which demonstrated that text-based prompts can guide pixel-level predictions without task-specific training. The few-shot adaptation idea extends classical few-shot learning paradigms (e.g., prototypical networks, matching networks) from classification to the denser task of segmentation, bridging the gap between one-shot learning literature and dense prediction. The paper likely benchmarks on standard OVS datasets (PascalVOC, COCO, or ADE20K splits) and shows quantitative improvements in mIoU over prior zero-shot and few-shot baselines. This work opens a research direction toward interactive or user-in-the-loop segmentation, where users can provide a handful of examples to refine model behavior on novel concepts in real time.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::