Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters
| Authors | Chris Vorster et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.04341 |
| Download | |
| Categories | cs.CV |
Abstract
In many CLIP adaptation methods, a blending ratio hyperparameter controls the trade-off between general pretrained CLIP knowledge and the limited, dataset-specific supervision from the few-shot cases. Most few-shot CLIP adaptation techniques report results by ablation of the blending ratio on the test set or require additional validation sets to select the blending ratio per dataset, and thus are not strictly few-shot. We present a simple, validation-free method for learning the blending ratio in CLIP adaptation. Hold-One-Shot-Out (HOSO) presents a novel approach for CLIP-Adapter-style methods to compete in the newly established validation-free setting. CLIP-Adapter with HOSO (HOSO-Adapter) learns the blending ratio using a one-shot, hold-out set, while the adapter trains on the remaining few-shot support examples. Under the validation-free few-shot protocol, HOSO-Adapter outperforms the CLIP-Adapter baseline by more than 4 percentage points on average across 11 standard few-shot datasets. Interestingly, in the 8- and 16-shot settings, HOSO-Adapter outperforms CLIP-Adapter even with the optimal blending ratio selected on the test set. Ablation studies validate the use of a one-shot hold-out mechanism, decoupled training, and improvements over the naively learnt blending ratio baseline. Code is released here: https://github.com/chris-vorster/HOSO-Adapter
Engineering Breakdown
Plain English
This paper addresses a critical practical problem in few-shot CLIP adaptation: the blending ratio hyperparameter that controls the trade-off between pretrained CLIP knowledge and new dataset-specific learning typically requires either test-set ablation or a separate validation set, violating the few-shot learning constraint. The authors introduce Hold-One-Shot-Out (HOSO), a validation-free method that learns the optimal blending ratio by holding out a single shot from the training set and using it to tune this hyperparameter. HOSO-Adapter achieves competitive results with existing methods while operating in a truly few-shot setting where no additional validation data or test-set tuning is required, making it more practical for real deployment scenarios.
Core Technical Contribution
The core novelty is a simple but effective validation-free hyperparameter selection strategy: rather than requiring an explicit validation set or test-set ablation, HOSO holds out one sample from the already-limited few-shot training set (e.g., 1 out of 4 shots in 4-shot learning) and uses that single held-out sample to estimate the optimal blending ratio. This transforms the blending ratio selection from a validation problem into an optimization problem solvable within the few-shot constraint. The key insight is that even a single held-out sample provides sufficient signal to make better blending ratio decisions than arbitrary defaults, enabling CLIP-Adapter-style methods to operate in the validation-free regime that matches the actual constraints of few-shot learning.
How It Works
HOSO operates within the CLIP-Adapter framework, which learns a lightweight adapter that blends predictions from the pretrained CLIP model with predictions from a few-shot-trained adapter. The blending is controlled by a scalar weight α that determines how much to trust each source. Standard CLIP-Adapter requires specifying α through test-set ablation (trying multiple values and picking the best on test data) or using a held-out validation set. HOSO instead reserves one sample from the N-shot training set; it trains the adapter on the remaining (N-1) shots, then evaluates multiple candidate α values on that single held-out shot and selects the α that minimizes loss on that sample. During final evaluation, the adapter is retrained on all N shots with the selected α. This approach requires only one additional forward-backward pass per candidate α value during selection, making it computationally tractable even with limited data.
Production Impact
This directly solves a real deployment constraint: in true few-shot scenarios (4-shot, 16-shot), engineers cannot afford to set aside a validation set without hurting downstream accuracy, yet most prior CLIP adaptation methods required this. HOSO eliminates the validation-set requirement, meaning teams can deploy few-shot adapted CLIP models without upfront data partitioning decisions or multiple experimental runs. The practical workflow becomes: collect your few-shot samples for a new domain, run HOSO-Adapter once, get a deployed model with the blending ratio optimized on actual few-shot data rather than external validation sets. The trade-off is minimal—you lose one training sample per dataset, which has negligible impact at 4-16 shots. Integration is straightforward since HOSO is additive to existing CLIP-Adapter implementations; production systems only need to implement the α selection loop and retrain with the selected value.
Limitations and When Not to Use This
The paper's scope is limited to CLIP-Adapter-style methods with blending ratio hyperparameters; it doesn't address blending in other few-shot CLIP architectures or other hyperparameters beyond the blending weight. The one-held-out sample may not provide stable estimates in extremely small data regimes (1-2 shots) or when the held-out sample is unrepresentative of the test distribution. The approach assumes the blending ratio that optimizes the held-out sample also optimizes test performance, which may not hold if the held-out sample is noisy or from a different domain-shift than the test set. The paper doesn't explore how HOSO scales to domains with extreme class imbalance or highly heterogeneous few-shot samples where a single sample may be an outlier.
Research Context
This work builds directly on CLIP-Adapter and the broader CLIP adaptation literature, which aims to efficiently fine-tune pretrained CLIP models for downstream tasks with minimal labeled data. It addresses a constraint that has been implicit in few-shot learning research but rarely made explicit: most few-shot papers do hyperparameter selection on test data or use external validation sets, which violates the spirit of few-shot learning. HOSO contributes to the emerging focus on validation-free learning in the few-shot regime, aligning with practical deployment requirements. The paper likely establishes or contributes to a new validation-free few-shot benchmark, pushing the field toward methods that are truly few-shot end-to-end without hidden data requirements.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::