Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models
| Authors | Md Tanvirul Alam |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.12119 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from rule-mapping failures. We study this behavior as semantic fixation: preserving a default interpretation even when the prompt specifies an alternative, equally valid mapping. To isolate this effect, we introduce VLM-Fix, a controlled benchmark over four abstract strategy games that evaluates identical terminal board states under paired standard and inverse rule formulations. Across 14 open and closed VLMs, accuracy consistently favors standard rules, revealing a robust semantic-fixation gap. Prompt interventions support this mechanism: neutral alias prompts substantially narrow the inverse-rule gap, while semantically loaded aliases reopen it. Post-training is strongly rule-aligned: training on one rule improves same-rule transfer but hurts opposite-rule transfer, while joint-rule training improves broader transfer. To test external validity beyond synthetic games, we evaluate analogous defamiliarization interventions on VLMBias and observe the same qualitative pattern. Finally, late-layer activation steering partially recovers degraded performance, indicating that semantic-fixation errors are at least partly editable in late representations. Project page, code, and dataset available at https://maveryn.github.io/vlm-fix/.
Engineering Breakdown
Plain English
This paper identifies and measures semantic fixation—a failure mode where large vision-language models stick to their default interpretation of a task even when given explicit instructions to use an alternative, equally valid rule system. The authors built VLM-Fix, a benchmark using four abstract strategy games (like chess or checkers variants) where they test the same board position under standard rules and inverse rules, allowing them to cleanly separate perception errors from rule-mapping failures. They tested 14 VLMs (both open-source and proprietary) and found a consistent accuracy gap favoring standard rules, demonstrating that models systematically fail to adopt new semantic mappings. Crucially, they showed that neutral prompt rewording significantly reduces this gap while semantically loaded prompts reopen it, suggesting the failure stems from learned associations rather than fundamental architectural limits.
Core Technical Contribution
The core contribution is isolating and quantifying semantic fixation as a distinct failure mode separate from perception errors—something previous evaluations conflated. The authors introduce VLM-Fix, a controlled benchmark methodology that uses paired rule formulations (standard vs. inverse) on identical board states to measure whether models can override default semantic priors when instructed. This is novel because it provides a clean experimental setup where perception is held constant and only the rule mapping changes, enabling researchers to measure the magnitude of the semantic fixation effect. The finding that prompt interventions (particularly neutral aliases) can substantially mitigate the gap suggests the failure is not a hard architectural constraint but rather a learnable bias that can be addressed through training or prompting strategies.
How It Works
The VLM-Fix benchmark works by creating pairs of rule formulations for four abstract strategy games, where one formulation is the standard interpretation (e.g., 'white pieces move forward, black pieces move backward') and the inverse applies opposite rules to the same board state ('white pieces move backward, black pieces move forward'). For each game, the authors generate identical terminal board positions and ask VLMs to evaluate them under both rule systems, testing whether the model can correctly apply the alternative mapping despite having learned the standard one. The key technical mechanism involves three layers of prompting variation: (1) baseline prompts with explicit rule statements, (2) neutral alias prompts that obscure the semantic connection (e.g., 'Group A' instead of 'white pieces'), and (3) semantically loaded prompts that reinforce default associations. By measuring accuracy deltas between standard and inverse rule conditions across these prompt variations, the authors quantify how much of the performance gap is attributable to hard perception failures versus recoverable semantic fixation. The results show that neutral aliases substantially recover performance on inverse rules, while semantically loaded language reintroduces the gap.
Production Impact
undefined
Limitations and When Not to Use This
The paper's evaluation is limited to abstract strategy games, which are highly structured, deterministic environments with well-defined rules and board states—this may not generalize to real-world visual domains with ambiguous interpretations, cluttered scenes, or continuous outcomes where semantic flexibility is less clearly defined. The benchmark tests only four games and doesn't establish how semantic fixation scales with task complexity, visual realism, or the number of alternative rule sets a model is expected to juggle, leaving open whether this is a small problem or a fundamental architectural constraint across all VLM applications. The paper doesn't provide theoretical analysis of why semantic fixation occurs at the representation level or offer principled methods to eliminate it during training; the prompt interventions are empirically effective but lack mechanistic grounding, so it's unclear whether the fixes address root causes or merely mask symptoms. Additionally, the evaluation doesn't assess whether addressing semantic fixation comes at a cost to standard-rule performance or how to balance flexibility with stability in production systems where reverting to defaults is sometimes the safe behavior.
Research Context
This work builds on a growing literature examining failure modes in large language and vision-language models beyond simple accuracy metrics—prior work identified issues like hallucination, reasoning shortcuts, and spurious correlations, but semantic fixation specifically addresses how models fail to override learned associations when instructed. The paper advances the evaluation methodology pioneered by benchmarks like MATH, CLEVR, and VQA-X by introducing the paired-rule protocol, which enables cleaner separation of failure sources and is a reusable technique for evaluating instruction following and rule adherence in multimodal models. It contributes to the broader safety and alignment literature by identifying a class of model failures (semantic stubbornness) that could affect real-world deployment in instruction-following systems, particularly in domains requiring flexible policy changes or multi-step adaptation. The finding that prompts can substantially mitigate the issue opens a research direction into efficient, training-free interventions and raises questions about whether semantic fixation is an inevitable consequence of scale and pretraining data distribution or an addressable inductive bias.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::