Counting to Four is still a Chore for VLMs
| Authors | Duy Le Dinh Anh et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.10039 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.
Engineering Breakdown
Plain English
This paper reveals that Vision-Language Models (VLMs) fundamentally struggle with basic object counting despite excelling at complex reasoning tasks. The authors built COUNTINGTRICKS, a controlled evaluation suite using simple geometric shapes to systematically expose where counting fails across different image patchification layouts and adversarial prompts. Through mechanistic analysis using attention patterns and component-wise probing, they discovered that counting-relevant visual information is actually captured in the modality projection stage but severely degrades as it flows through subsequent layers of the model. This finding is surprising because it shows the problem isn't missing the visual signal—it's losing it during processing.
Core Technical Contribution
The core contribution is a mechanistic understanding of VLM counting failures through both behavioral and internal-state analysis. Rather than just measuring final accuracy on counting tasks, the authors developed a diagnostic methodology that traces where relevant visual evidence exists and where it disappears inside the model architecture. COUNTINGTRICKS serves as a precision diagnostic tool—using controlled shape-based inputs and systematic adversarial conditions to isolate failure modes in ways that standard benchmarks cannot. The key insight is that the failure isn't in vision encoding or language generation separately, but specifically in how the visual signal propagates and is used through the multimodal fusion and reasoning layers.
How It Works
The evaluation starts with COUNTINGTRICKS, a controlled suite of simple geometric counting scenarios where the ground truth is mathematically certain (count shapes on an image). Inputs vary systematically across patchification layouts (how the image is tokenized/divided into patches) and adversarial prompting conditions to stress-test robustness. The authors run inference through the VLM while simultaneously capturing internal representations: attention maps showing which image patches the model attends to, and component-wise probes that measure how much counting information is present at each layer. They trace the signal forward through the pipeline—from initial vision encoding through modality projection (where visual embeddings are aligned with language embeddings) through the transformer reasoning layers to final token prediction. By comparing where counts are 'correct' internally versus where the final output fails, they pinpoint exactly which architectural stage loses counting information.
Production Impact
For teams deploying VLMs in production systems requiring visual grounding (inventory management, quality control, document understanding), this work provides concrete diagnostics for when and why counting failures will occur. Before adopting a VLM for counting-critical applications, engineers can now use COUNTINGTRICKS to benchmark performance under realistic conditions including different image tokenization schemes that may occur in their deployment pipeline. The findings suggest that simply fine-tuning or prompt engineering on the final layer won't fix counting—the issue is architectural, meaning teams may need to either accept counting limitations, retrain components specifically for numerical reasoning, or use specialized external modules for tasks requiring precise counts. The mechanistic insights enable targeted interventions: if degradation happens in modality projection, engineers know to focus calibration efforts there rather than wasting resources on output layer tuning. However, applying these diagnostics requires access to model internals (attention weights, hidden states), which may not be available in some closed-source VLM APIs.
Limitations and When Not to Use This
This analysis is restricted to simple geometric counting tasks in controlled settings—the findings may not fully explain failures on naturalistic images with overlapping objects, occlusion, or complex textures. The paper doesn't provide solutions, only diagnosis of the problem, so teams still face the challenge of actually fixing VLMs to improve counting rather than just understanding why they fail. The mechanistic probing approach depends on having white-box access to model internals, making these diagnostics unavailable for black-box API-based deployments of proprietary models. Additionally, the controlled nature of COUNTINGTRICKS may not expose failure modes that only emerge in the wild—adversarial conditions in the evaluation may not match real-world distribution shifts or the specific adversarial prompts users will naturally try.
Research Context
This work builds on a growing body of research showing that despite benchmark success, VLMs have fundamental limitations on grounding tasks—prior work documented failures in spatial reasoning and visual understanding despite strong performance on VQA benchmarks. The mechanistic analysis methodology aligns with recent interpretability research (like those investigating attention patterns in transformer models) but applies it specifically to the multimodal failure case. This opens a research direction toward understanding not just whether VLMs fail but why, which is essential for the community to develop architecturally-informed solutions rather than patch-based fixes. The paper contributes to building a more nuanced understanding of VLM capabilities—moving beyond aggregate metrics to component-level diagnostics that will drive the next generation of vision-language architecture improvements.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::