Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
| Authors | Amita Kamath et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23351 |
| Download | |
| Categories | cs.CL, cs.CV |
Abstract
The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.
Engineering Breakdown
Plain English
This paper identifies why Vision-Language Models (VLMs) like OpenCLIP, LLaVA-1.5, and Molmo struggle with reasoning tasks despite being trained on web-scale data. The authors discovered that the problem isn't lack of data volume, but rather reporting bias in how humans naturally describe images—people say 'at the game!' instead of 'a photo of 37 people standing behind a field,' omitting the quantitative and spatial details needed to learn reasoning. By analyzing their training data through pragmatic theory from linguistics, they found critical gaps in four core reasoning skills: spatial reasoning, temporal reasoning, negation, and counting. The paper demonstrates that even synthetic data doesn't fully overcome this bias, meaning scale alone cannot fix reasoning capabilities without addressing the underlying data annotation practices.
Core Technical Contribution
The core novelty is applying pragmatic theory—specifically the concept of reporting bias from human communication—to diagnose reasoning failures in VLMs. Rather than proposing a new model architecture or training method, the authors provide a principled framework for understanding why existing models fail at specific reasoning tasks by analyzing what information is systematically absent from training captions. They conducted an empirical audit of popular VLM training corpora and quantified the under-representation of four specific reasoning modalities. This reframes the reasoning problem from a capability gap to a data representation problem, suggesting that solutions require richer annotation practices rather than simply scaling existing approaches.
How It Works
The investigation proceeds in three phases: first, the authors extract and analyze captions from training datasets used by OpenCLIP, LLaVA-1.5, and Molmo to measure how frequently spatial, temporal, negation, and counting expressions appear relative to how often these concepts are visually present. Second, they apply pragmatic theory to explain why humans naturally under-report such information—human communication omits obvious or context-dependent details, a phenomenon called reporting bias. Third, they create curated benchmarks to measure reasoning performance gaps and correlate those gaps with the measured under-representation in training data. The key insight is that the model's reasoning failures directly trace to missing supervision signals in the training data rather than architectural limitations, and even web-scale datasets don't fix this because the bias is inherent to how humans communicate rather than a sampling problem.
Production Impact
For teams building VLM-based systems that require reasoning (spatial layout understanding, object counting, temporal sequencing, negation handling), this paper signals that scaling existing datasets won't reliably improve performance. You'd need to actively curate or augment training data to include reasoning-focused captions—either through human annotation with explicit guidelines to include quantitative and spatial details, or through synthetic caption generation that oversamples these modalities. This could increase annotation costs significantly (moving from passive web-scraping to active curation) but would provide more reliable performance on downstream reasoning tasks like visual question answering or scene understanding. In production, you might implement a data audit pipeline similar to the authors' analysis to identify which reasoning modalities your models are weak on, then prioritize rebalancing your training distribution accordingly. The trade-off is higher data preparation overhead but potentially better generalization on reasoning-heavy tasks without requiring larger models.
Limitations and When Not to Use This
The paper doesn't propose a complete solution—it diagnoses the problem but notes only that 'curated' data helps, without fully specifying what a complete remedy looks like or how much synthetic/curated data is needed to fully close the gap. The analysis is limited to four reasoning skills and doesn't address other types of reasoning (causal, commonsense, functional) that VLMs also struggle with. The work assumes that reporting bias is the primary cause of reasoning failures, but doesn't fully disentangle this from other potential factors like model capacity, optimization objectives (contrastive learning), or architectural design choices. Finally, the paper doesn't provide concrete recipes for practitioners—there's no released dataset, no benchmark with standard splits, and no empirical validation showing that following the authors' recommendations actually solves the reasoning problem at scale in production settings.
Research Context
This work builds on a long line of research showing that VLMs struggle with structured reasoning and fine-grained visual understanding, but shifts the blame from model capability to data quality. It draws on pragmatics literature (Grice's maxims of communication) to provide a linguistic framework for understanding dataset biases, connecting vision-language research with insights from cognitive science and linguistics. The paper contributes to an emerging area of work auditing and improving training data for large vision-language models—related to recent efforts in data-centric AI that emphasize data quality over model scale. It opens a research direction toward 'reasoning-aware' dataset curation practices and suggests that future VLM progress may depend less on architectural innovations or massive scale and more on thoughtful engineering of what information is present in training annotations.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::