Large Multimodal Models as General In-Context Classifiers
| Authors | Marco Garosi et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.23229 |
| Download | |
| Categories | cs.CV |
Abstract
Which multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive Vision-Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We benchmark state-of-the-art LMMs on diverse datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their "in-context" equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.
Engineering Breakdown
Plain English
This paper challenges the conventional wisdom that CLIP-like contrastive models are always best for image classification by demonstrating that Large Multimodal Models (LMMs) can match or exceed CLIP's performance when given a few in-context examples, without requiring fine-tuning or adapter caching. The authors benchmark state-of-the-art LMMs across diverse closed-world classification datasets and show that while LMMs underperform CLIP in pure zero-shot settings, adding just a handful of labeled examples enables LMMs to reach or surpass CLIP with cache-based adapters. They extend their analysis to open-world classification where the generative nature of LMMs provides advantages that contrastive models cannot match. The core finding is that in-context learning capability—a strength of instruction-tuned LMMs—is a powerful and underexploited tool for classification that rivals specialized contrastive approaches.
Core Technical Contribution
The paper's central contribution is empirically demonstrating and formalizing that Large Multimodal Models possess a practical in-context learning capability for classification that has been largely overlooked in favor of zero-shot CLIP performance. Rather than treating LMMs as tools only for complex reasoning tasks, the authors show that LMMs can function as general-purpose classifiers that gracefully degrade with fewer examples but improve substantially with in-context demonstrations—often matching or exceeding the performance of CLIP with cache-based adapters that also use in-context examples. The technical novelty lies in the systematic evaluation showing LMMs don't need architectural modifications or fine-tuning to be competitive; the generative parameterization combined with instruction-tuning already enables strong few-shot classification. This reframes LMMs from specialized reasoning engines to general-purpose classification systems where the number of in-context examples acts as a simple, interpretable control lever for accuracy-efficiency trade-offs.
How It Works
The approach evaluates LMMs by constructing in-context learning prompts where the model receives a query image alongside K labeled example images and their class labels, then generates the predicted class. For each test image, the LMM processes the multimodal input (images plus text instructions describing the classification task) and produces a class prediction, typically by constraining the output to valid class names or probability distributions. The key mechanism is that the LMM's instruction-following and few-shot reasoning capabilities—learned during pretraining on diverse multimodal data—allow it to infer classification patterns from the provided examples without any weight updates. The comparison baseline is CLIP with cache-based adapters, which similarly use in-context examples but operate in the contrastive embedding space rather than through generative token prediction. The evaluation metric is accuracy across diverse closed-world and open-world classification benchmarks, with the number of in-context examples (K) varied to measure how performance scales. The paper also likely includes ablations on prompt formulation, example selection strategies, and model size to isolate which factors drive performance gains.
Production Impact
For engineers building production classification systems, this work suggests that deploying a single large multimodal model can replace a two-model setup (CLIP for zero-shot + specialized classifier for few-shot adaptation). The practical implication is substantial: instead of maintaining separate inference pipelines for different data regimes, teams can use one LMM that automatically improves when a small labeled dataset becomes available, with no retraining or adapter caching required. This simplifies the deployment pipeline and reduces inference latency since you avoid the overhead of CLIP+adapter ensemble or cache lookups. The trade-off is that LMM inference is more compute-intensive per query than CLIP (larger model, longer token sequences for in-context examples), so organizations with strict latency budgets (e.g., mobile or real-time systems) may still prefer CLIP's efficiency. The approach also shifts the optimization burden from model selection and fine-tuning to prompt engineering and example selection—teams need to invest in choosing which examples to include in context, which adds a new hyperparameter tuning surface but eliminates expensive retraining cycles when class distributions or label availability change.
Limitations and When Not to Use This
The paper's scope is limited to closed-world and open-world classification, so it does not address more complex vision tasks like detection, segmentation, or grounding where bounding box or pixel-level outputs are required—LMMs may not maintain the same advantage there. The in-context learning approach requires passing multiple images through the model for every prediction, which scales poorly with context length and batch size; the paper likely does not fully explore the computational penalty or latency impact of this compared to CLIP's single-forward-pass efficiency for many images. The work assumes that the in-context examples are well-chosen and representative; performance could degrade significantly with noisy, unbalanced, or non-representative examples, and the paper may not thoroughly characterize robustness to poor example selection. Additionally, the comparison to CLIP assumes cache-based adapters as the baseline for in-context CLIP; it's unclear whether other VLM adaptation methods (fine-tuning, LoRA, prompt learning) might close the gap or exceed the LMM approach, limiting the generality of conclusions about which model family is best.
Research Context
This work builds on the foundational insight that vision-language models trained with contrastive objectives (CLIP) excel at zero-shot transfer due to their aligned embedding space, but extends the narrative by highlighting that newer instruction-tuned LMMs (GPT-4V, LLaVA, etc.) have their own underutilized strength in few-shot in-context learning. The paper sits at the intersection of two research directions: the rise of large language models' in-context learning capabilities (demonstrated in works like GPT-3) and the application of multimodal models to vision tasks traditionally dominated by CLIP. It challenges the benchmark-driven narrative where zero-shot performance is the primary metric, arguing that practical systems often have access to a few labeled examples and should optimize for that regime instead. The work opens a new research direction asking: how can we design LMMs and prompting strategies specifically optimized for in-context classification? And what other vision tasks (detection, segmentation) could benefit from the in-context learning paradigm that LMMs offer?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::