Multimodal Large Language Models as Image Classifiers
| Authors | Nikita Kisel et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06578 |
| Download | |
| Categories | cs.CV |
Abstract
Multimodal Large Language Models (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and vision-language models report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and an open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLMs underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLMs predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation.
Engineering Breakdown
Plain English
This paper investigates why different studies reach conflicting conclusions when comparing Multimodal Large Language Models (MLLMs) to traditional supervised and vision-language models for image classification. The authors identified three critical evaluation protocol issues: model outputs outside the provided class list being discarded, inflated accuracy from weak multiple-choice options, and open-world settings that fail due to poor output mapping rather than actual model limitations. They also quantify how overlooked design choices like batch size, image ordering, and text encoder selection substantially impact measured accuracy. By fixing these protocol issues, the paper provides a more reliable framework for benchmarking MLLMs and shows that performance conflicts in prior work stem from evaluation methodology rather than fundamental model capability differences.
Core Technical Contribution
The core contribution is a systematic audit of MLLM evaluation protocols that reveals methodological artifacts inflating or deflating performance measurements. Rather than proposing a new architecture or algorithm, the authors identify and fix specific evaluation bugs: handling out-of-vocabulary model predictions, controlling for distractor difficulty in multiple-choice setups, and proper output mapping in open-world scenarios. They also quantify the impact of previously overlooked hyperparameter choices (batch size, image ordering, text encoder) that significantly affect measured accuracy but aren't typically reported. This evaluation-focused contribution is novel because it shows that conflicting conclusions in prior MLLM comparison studies aren't due to fundamental differences in model capability but rather inconsistent and problematic evaluation protocols.
How It Works
The paper's approach involves designing a unified evaluation framework for MLLMs used as image classifiers with multiple variants. For input, they take standard image classification datasets and evaluate MLLMs by constructing prompts that ask the model to classify images into provided categories. The key technical mechanism involves several fixes: (1) tracking when model outputs don't match any class in the ground truth list and handling these appropriately rather than discarding them, (2) controlling the quality and similarity of multiple-choice distractors so that inflated accuracy from easy guessing is prevented, and (3) implementing proper output mapping functions that ensure model predictions are correctly mapped to class labels even when the model uses synonyms or variations. The paper then systematically ablates design choices by varying batch size during inference, reordering images in different ways, and swapping text encoders (e.g., different CLIP variants), measuring how each factor impacts final accuracy. The evaluation compares MLLMs against supervised models and vision-language models under these standardized, corrected protocols to produce more reliable relative performance measurements.
Production Impact
For engineers deploying MLLMs in classification pipelines, this paper directly impacts benchmarking and model selection decisions. In production, you would need to implement the protocol fixes described: robust output parsing that maps model text generations to actual class labels (handling synonyms and out-of-distribution outputs), careful distractor selection for validation datasets, and proper handling of confidence scores rather than discarding uncertain predictions. The quantification of design choice impact is immediately actionable—batch size and image ordering appear to significantly affect accuracy, meaning you must standardize these during evaluation if you want reproducible results and fair model comparisons. The main trade-off is increased evaluation complexity and cost: proper protocol implementation requires more careful dataset construction, output post-processing logic, and ablation studies during model selection. However, this investment prevents costly mistakes like selecting an inferior model based on inflated benchmark results or dismissing capable models due to unfair evaluation.
Limitations and When Not to Use This
The paper focuses on evaluation methodology but doesn't propose algorithmic improvements to MLLMs themselves—it shows what existing models can do fairly measured rather than how to make them better at classification. The fixes assume access to well-defined closed-set class hierarchies, which limits applicability to open-vocabulary or truly open-world scenarios where the class set is unknown or constantly expanding. The study doesn't address domain-specific challenges like out-of-distribution detection or calibrated confidence estimation, which are critical for production systems that need to know when predictions are unreliable. Additionally, while the paper quantifies the impact of batch size and image ordering, it doesn't provide principled guidance on optimal choices for these hyperparameters—the findings appear empirical and may not transfer to different model architectures or datasets.
Research Context
This work builds on the rapid emergence of MLLMs (like LLaVA, GPT-4V, Gemini) and grows from the broader concern that benchmark results are inconsistent across papers. It intersects with the evaluation methodology literature in machine learning, which has repeatedly documented how protocol choices dramatically affect reported performance (similar to discussions around data leakage, train-test overlap, and benchmark saturation). The paper contributes to the vision-language model research direction by clarifying what these models actually achieve on standard benchmarks when fairly evaluated. This work is particularly relevant as MLLMs become commercially deployed and practitioners need reliable guidance on whether to use them versus task-specific supervised or vision-language models—the paper provides that grounding by removing methodological noise from the comparison.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::