Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models
| Authors | Yunkai Zhang et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2604.09687 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap Digital Agnosia. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.
Engineering Breakdown
Plain English
This paper introduces Grid2Matrix (G2M), a benchmark that tests whether Vision-Language Models can faithfully capture all visual details in an image by having them read color grids and convert them to numbered matrices based on a provided color-to-number mapping. The researchers found that VLMs fail dramatically and suddenly on surprisingly small grids rather than degrading gradually as complexity increases—a phenomenon they call "digital agnosia" (inability to read visual information). By systematically varying grid size and color count, G2M isolates visual perception failures from semantic reasoning confounds that hide in standard benchmarks. The paper also probes the visual encoders of leading VLM families to identify where exactly these failures originate.
Core Technical Contribution
The core contribution is Grid2Matrix itself: a minimalist, controlled evaluation benchmark that forces exhaustive visual readout without relying on semantic reasoning or common-sense knowledge. Unlike existing VLM benchmarks (which allow partial image understanding and high-level reasoning to mask perceptual failures), G2M requires pixel-perfect fidelity—every grid cell must be correctly identified and mapped. The benchmark reveals a sharp, early-collapse failure mode in zero-shot performance that doesn't appear in standard evals, exposing a fundamental gap between how well VLMs appear to perform and how faithfully they actually process visual input. This is not a new architecture or training method, but rather a diagnostic tool that exposes blind spots in current VLMs through systematic variation of visual complexity.
How It Works
The G2M benchmark operates as follows: (1) A VLM is presented with a colored grid image where each cell has a distinct or repeated color. (2) The model is also given a color-to-number mapping (e.g., red→1, blue→2, green→3). (3) The model must output a matrix where each cell contains the corresponding number for its color. (4) By varying grid dimensions (e.g., 4×4, 8×8, 16×16) and the number of distinct colors (e.g., 2 colors, 5 colors, 10 colors), the researchers systematically increase visual complexity while keeping the task semantically trivial. (5) Performance is measured as exact-match accuracy on the final matrix output. The researchers then ablate and probe the visual encoders (the first-stage component that converts raw pixels into embeddings) of representative VLM families to isolate whether failures occur at encoding, multi-modal fusion, or language generation stages.
Production Impact
For engineers building production VLM systems, this work has immediate diagnostic value: it reveals that standard benchmarks (MMVP, LLaVA, etc.) can mask serious perceptual failures because they tolerate partial image understanding. If you're deploying a VLM for any task requiring exhaustive visual readout—document OCR, table extraction, visual inspection in manufacturing, or reading annotations on technical diagrams—you should validate it against a Grid2Matrix-like test before production. The sharp early-collapse finding means you can't rely on gradual degradation as a safety signal; performance will drop off a cliff at surprisingly small problem sizes, so you need explicit failure-mode testing. The paper's encoder-level analysis tells you where to focus optimization: if the bottleneck is in the visual encoder (as the paper suggests), improvements there will have wider impact than decoder tuning alone.
Limitations and When Not to Use This
Grid2Matrix, while revealing, is an artificial task that may not predict real-world performance on more semantically rich visual problems where models can leverage context and reasoning to compensate for encoding gaps. The benchmark does not provide guidance on how to fix the identified failures—it diagnoses the problem but doesn't offer algorithmic solutions for improving visual fidelity. The paper tests only current-generation VLMs and may not generalize to future architectures (e.g., those with higher-resolution encoders or fundamentally different fusion strategies). Additionally, G2M's grid format is highly structured and uniform; performance on this task may not correlate with failure modes on natural images, cluttered scenes, or images with perspective distortion, which present different perceptual challenges.
Research Context
This work builds on a growing line of research questioning whether VLMs truly understand visual content or merely exploit high-level semantic patterns (prior work includes probing studies on VLM hallucination and fine-grained visual understanding). It extends benchmarking methodology from standard evals (COCO, VQA, TextVQA) toward controlled, diagnostic evaluation that minimizes confounds—similar in philosophy to adversarial or unit-test approaches in vision. The paper contributes to the safety and interpretability subfield by making explicit a failure mode (digital agnosia) that was previously implicit in scattered performance gaps. The Grid2Matrix framework opens the door to systematic study of visual encoding capacity and could inspire follow-up work on scaling visual resolution, improving spatial reasoning, or testing multi-modal alignment in more controlled settings.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::