Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe
| Authors | Chris Vorster et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.04346 |
| Download | |
| Categories | cs.CV |
Abstract
Large-scale Vision-Language Foundation Models (VLFMs), such as CLIP, now underpin a wide range of computer vision research and applications. VLFMs are often adapted to various domain-specific tasks. However, VLFM performance on novel, specialised, or underrepresented domains remains inconsistent. Evaluating VLFMs typically requires labelled test sets, which are often unavailable for niche domains of interest, particularly those from the Global South. We address this gap by proposing a highly data-efficient method to predict a VLFM's zero-shot accuracy on a target domain using only a single labelled image per class. Our approach uses a Large Language Model to generate plausible counterfactual descriptions of a given image. By measuring the VLFM's ability to distinguish the correct description from these hard negatives, we engineer features that capture the VLFM's discriminative power in its shared embedding space. A linear regressor trained on these similarity scores estimates the VLFM's zero-shot test accuracy across various visual domains with a Pearson-r correlation of 0.96. We demonstrate our method's performance across five diverse datasets, including standard benchmark datasets and underrepresented datasets from Africa. Our work provides a low-cost, reliable tool for probing VLFMs, enabling researchers and practitioners to make informed decisions about data annotation efforts before committing significant resources. The model training code, generated captions and counterfactuals are released here: https://github.com/chris-vorster/PreLabellingProbe.
Engineering Breakdown
Plain English
This paper addresses a critical gap in evaluating Vision-Language Foundation Models (VLFMs) like CLIP on niche or underrepresented domains where labeled test data doesn't exist. The authors propose a one-shot probe method that predicts a VLFM's zero-shot accuracy on a target domain using only a single labeled image per class, avoiding the need to collect expensive domain-specific datasets. The approach leverages a Large Language Model to generate counterfactual descriptions of images, then measures how well the VLFM can distinguish correct descriptions from plausible alternatives. This enables practitioners to quickly assess whether a foundation model will work for specialized tasks—particularly important for domains in the Global South and other underrepresented areas—without requiring labeled validation sets.
Core Technical Contribution
The core innovation is a data-efficient probing method that decouples foundation model evaluation from the requirement of large labeled test sets. Instead of traditional zero-shot or few-shot adaptation benchmarks, the authors use LLM-generated counterfactual descriptions as synthetic negatives, transforming the problem into measuring model discrimination ability from a single example. This approach makes foundation model assessment feasible for resource-constrained scenarios and niche domains where collecting representative test data is impractical. The key insight is that a single image per class, combined with LLM-generated plausible but incorrect alternatives, provides sufficient signal to predict whether a foundation model has learned the necessary visual-semantic alignments for that domain.
How It Works
The method operates in three stages: (1) Given a target domain and a single labeled image per class, the authors extract visual features from the image using the VLFM's vision encoder. (2) An LLM generates multiple plausible counterfactual text descriptions—semantically related but incorrect labels that a typical vision-language model might confuse with the correct label. For example, for a medical imaging task, if the correct label is 'adenocarcinoma,' the LLM might generate 'squamous cell carcinoma' or 'benign lesion' as counterfactuals. (3) The VLFM's text encoder scores both the correct and counterfactual descriptions against the image's visual features, computing a discrimination score or ranking metric. Higher discrimination accuracy between correct and counterfactual descriptions predicts better zero-shot performance on the actual domain. This single-shot scoring avoids the need for a separate evaluation dataset while remaining computationally efficient.
Production Impact
For engineers deploying foundation models to specialized domains, this method provides a lightweight pre-deployment assessment tool that answers 'will CLIP work for my medical imaging/satellite/autonomous vehicle application?' without requiring months of data collection and annotation. Integration into production pipelines is straightforward: add a validation step that takes one labeled example per target class, generates counterfactuals via an LLM API call, and runs inference through the VLFM to get a prediction score before committing to fine-tuning or full evaluation. The approach dramatically reduces time-to-assessment for niche domains—instead of 2-3 weeks building a test set, you get signal in hours. The main trade-off is that you're measuring proxy metrics (discrimination ability) rather than true task performance, so results should be validated on at least a small holdout set if the stakes are high; additionally, LLM quality and domain knowledge directly affect counterfactual quality, introducing a dependency on the LLM's capabilities.
Limitations and When Not to Use This
The method assumes that LLM-generated counterfactuals accurately reflect the domain's actual confusion space—if the LLM doesn't understand domain-specific subtleties or generates implausible alternatives, the probe signal degrades. The approach relies on single examples per class, which may not capture intra-class variance or rare edge cases that matter in production; a domain with high visual diversity within each class could yield misleading predictions. The paper doesn't address how to select which single image to use for each class when you have some labeled data, or how sensitive the method is to image selection bias. Additionally, the work focuses on zero-shot or minimal fine-tuning scenarios; it's unclear how well this probe predicts performance when you do have resources for substantial fine-tuning or when dealing with completely out-of-distribution domains where even the LLM has weak priors. Follow-up work should explore robustness to LLM quality, comparison against random or adversarial counterfactuals, and validation across broader domain gaps.
Research Context
This work builds on the growing recognition that foundation model evaluation is expensive and the push toward data-efficient model assessment techniques, extending the zero-shot CLIP evaluation paradigm to resource-constrained settings. It connects to research on probe classifiers, model-agnostic evaluation metrics, and the underrepresentation problem in ML (where models trained on large Western datasets fail on Global South domains). The paper is positioned at the intersection of foundation model adaptation, few-shot learning, and practical ML deployment, directly addressing a real bottleneck in democratizing foundation models for specialized domains. It opens research directions around synthetic evaluation datasets generated by LLMs, cost-efficient validation for foundation models, and domain-specific bias measurement without full labeled evaluations.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::