Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
| Authors | Yige Xu et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.16256 |
| Download | |
| Categories | cs.CV, cs.CL |
Abstract
Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.
Engineering Breakdown
Plain English
This paper introduces CrossMath, a new benchmark for measuring whether vision-language models (VLMs) actually reason about images or just rely on their text processing abilities. The researchers built problems in three formats—text-only, image-only, and image+text—with identical task-relevant information verified by humans, creating a controlled setup to isolate vision-specific reasoning. This benchmark directly addresses a critical gap: it's unclear whether VLMs' strong performance comes from genuine multimodal reasoning or primarily from the reasoning capabilities of their underlying language models. The controlled comparison methodology eliminates confounding factors like information format differences, enabling precise measurement of modality-specific contributions.
Core Technical Contribution
The core novelty is the CrossMath benchmark's rigorous multi-modal alignment methodology. Unlike existing VLM benchmarks that conflate image understanding with text reasoning, CrossMath guarantees that identical task-relevant information appears across text-only, image-only, and image+text conditions, verified by human annotators. This design isolates vision-grounded reasoning by creating controlled cross-modal comparisons where the only variable is modality presentation, not information content. The contribution is fundamentally about measurement and evaluation—providing a principled framework to disentangle multimodal reasoning from unimodal language model reasoning, which prior benchmarks could not do.
How It Works
The benchmark construction process begins with problem generation where each problem is formulated in three parallel conditions: text-only format containing full task information, image-only format with the visual representation, and image+text combining both modalities. Human annotators then verify that task-relevant information is truly identical across all three versions, eliminating confounds where, for example, text descriptions might accidentally provide hints not present in images. Test instances are presented to VLMs in each condition, and performance is measured separately—text-only provides a language-model baseline, image-only measures vision capability, and image+text shows combined performance. By comparing performance deltas across conditions (image+text minus text-only vs. image-only performance), researchers can quantify whether the VLM's vision integration actually improves reasoning or whether text reasoning dominates. The metric design specifically isolates vision-grounded reasoning contribution by controlling information availability.
Production Impact
For engineers deploying VLMs in production, CrossMath provides a diagnostic framework to validate whether your system actually leverages vision or just reads text descriptions. If a VLM shows minimal performance improvement from images compared to text-only baselines, you've learned that investing in high-quality image preprocessing may not help—the bottleneck is language reasoning, not vision. This directly impacts architecture decisions: should you invest in better visual encoders, or is the language backbone the limiting factor? In multimodal applications like document understanding, visual question answering, or medical imaging analysis, this benchmark helps you measure modality-specific contributions before deployment. The trade-off is that generating CrossMath-style benchmarks requires human annotation to verify information equivalence across modalities, adding development overhead but providing high-confidence performance diagnostics.
Limitations and When Not to Use This
The paper's scope appears limited to mathematical/reasoning tasks based on the 'CrossMath' name, so it's unclear whether findings generalize to other VLM applications like image classification, detection, or open-ended visual understanding. The requirement for human verification of information equivalence across modalities is labor-intensive and may not scale to diverse task domains—what counts as 'identical task-relevant information' is subjective and task-dependent. The benchmark assumes that modality contributions are additive and independent, which may not hold if image and text information interact synergistically or if certain reasoning types fundamentally require both modalities. Additionally, the paper doesn't address how to construct similar benchmarks for domains where text and image information cannot be made strictly equivalent (e.g., visual aesthetics, spatial reasoning, temporal reasoning in videos), limiting the framework's broader applicability.
Research Context
This work builds on a growing concern in VLM evaluation: existing benchmarks like MMVP, LLAVA, and Flamingo don't isolate modality contributions, so researchers cannot distinguish whether performance comes from vision reasoning or language model reasoning. The paper addresses a fundamental question raised by recent work on language model scaling and multimodal emergent abilities—namely, whether VLMs exhibit genuine cross-modal reasoning or language-centric behavior. CrossMath extends the evaluation methodology lineage from prior diagnostic benchmarks (like those testing reasoning shortcuts in vision models) into the multimodal setting. This research direction opens doors for developing modality-specific diagnostic suites, potentially inspiring similar controlled frameworks for other modalities (audio-language, video-language) and contributing to more rigorous VLM evaluation standards across the field.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::