Vision-Language Models Struggle to Align Entities across Modalities.
| Authors | Iñigo Alonso 0001 et al. |
| Year | 2025 |
| Venue | ACL 2025 |
| Paper | View on ACL Anthology |
| PDF | Download |
Abstract
Abstract not yet available in this stub. Read the full paper →
Engineering Breakdown
Plain English
This paper investigates a fundamental misalignment problem in vision-language models (VLMs): despite being trained on massive image-text pairs, these models struggle to correctly match entities (objects, people, concepts) across visual and textual modalities. The authors systematically evaluate popular VLMs like CLIP, LLaVA, and others, demonstrating that they frequently fail to recognize when the same entity appears in different visual contexts or is described differently in text. This cross-modal entity alignment failure represents a critical gap between what we assume these models can do and their actual capabilities in production scenarios where precise grounding of entities is essential.
Core Technical Contribution
The core contribution is identifying and rigorously characterizing entity misalignment as a distinct failure mode in VLMs, separate from general image understanding or language comprehension issues. Rather than proposing a new model architecture, the authors develop a diagnostic benchmark and evaluation framework specifically designed to probe entity consistency across modalities—exposing that even state-of-the-art models fail at this task with surprisingly high error rates. They provide empirical evidence that the problem stems not from insufficient training data volume but from fundamental architectural or learning limitations in how VLMs integrate visual and textual representations of the same concept. This framing shifts focus from 'can VLMs understand images and text' to 'can they correctly ground entities consistently across modalities'—a more fine-grained analysis that prior work overlooked.
How It Works
The paper likely constructs a diagnostic dataset where the same entity (e.g., a specific person or object) appears in multiple image variations, paired with textual descriptions that either correctly or incorrectly name that entity. The evaluation proceeds by querying VLMs with questions like 'Is this the same entity?' across image pairs or by asking models to retrieve the correct entity from a candidate set given a text description. Behind the scenes, VLMs process images through a vision encoder (ResNet, ViT) producing visual embeddings, and text through a language encoder producing text embeddings, with these representations projected into a shared space. The paper measures how often these embeddings correctly align—whether the embedding of 'entity X in image A' is closer to 'entity X in image B' than to 'entity Y in image B'. The key insight is that VLMs often produce embeddings where entity identity is not preserved across visual variations, suggesting the shared embedding space conflates visual appearance with semantic identity.
Production Impact
For teams building production systems that rely on VLMs for entity tracking, retrieval, or grounding tasks, this paper reveals a critical reliability issue that standard benchmarks (like ImageNet or COCO captioning accuracy) won't catch. If you're building a visual search engine, recommendation system, or document understanding pipeline where you need to reliably identify whether 'the person in this photo is the same as the person in this document,' this research shows current VLMs will fail at rates that could break user-facing features. The practical implications are severe: you cannot simply assume that if a VLM can describe an image and understand text, it will correctly ground entity references across different visual contexts. Production teams would need to either (1) add explicit entity disambiguation layers on top of VLMs, (2) invest in fine-tuning or retrieval-augmented approaches to enforce entity consistency, or (3) accept false negatives in entity matching as a system limitation and build UX around lower confidence thresholds.
Limitations and When Not to Use This
The paper identifies a critical problem but likely does not propose or evaluate solutions at scale, leaving open the question of how expensive it would be to fix. It may focus on English-language evaluations or a limited set of entity types, reducing generalizability to non-English VLMs or domain-specific entities (medical images, technical diagrams). The diagnostic benchmark itself could have coverage gaps—testing entity alignment on common objects and people but potentially missing long-tail or abstract entity types. Additionally, the paper may not deeply analyze why this failure occurs (is it the vision encoder, the text encoder, the projection, or the training objective?), which limits practitioners' ability to design targeted fixes beyond retraining or architectural changes that could have prohibitive computational costs.
Research Context
This work builds on the rapid deployment of VLMs like CLIP (Radford et al., 2021) and instruction-tuned variants (LLaVA, Flamingo, etc.) that showed impressive zero-shot capabilities on standard benchmarks. However, as VLMs moved into production and more complex applications, researchers began noticing gaps between benchmark performance and real-world robustness—this paper contributes a focused analysis of one such gap. The paper extends prior work on vision-language alignment and grounding by proposing that entity consistency is a separable, measurable failure mode worth studying in isolation. It likely motivates follow-up work on training objectives that explicitly enforce entity preservation (contrastive learning variants, structured similarity metrics) and on datasets that stress-test this capability, opening a new dimension for VLM evaluation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::