Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
| Authors | Xinru Yan et al. |
| Year | 2026 |
| HF Upvotes | 6 |
| arXiv | 2604.16902 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference
Engineering Breakdown
Plain English
This paper investigates a critical but overlooked issue in native Omni-modal Large Language Models (OLLMs) that process multiple types of data in a unified way: these models exhibit strong preferences for certain modalities over others. The authors built a benchmark based on conflicting information across modalities and measured modality preference using a metric called modality selection rate, evaluating ten leading OLLMs. They found a surprising shift from traditional vision-language models (VLMs) that favor text to most OLLMs strongly preferring visual information. Through layer-wise probing, they discovered this preference isn't hardwired but emerges gradually as information flows through the middle and later layers of the model.
Core Technical Contribution
The core novelty is systematic characterization of modality preference in native OLLMs, which has been largely ignored despite being fundamental to understanding model behavior. The authors introduce a conflict-based benchmark specifically designed to surface cases where different modalities provide contradictory information, forcing the model to choose which to trust. They develop the modality selection rate metric to quantify this preference in a measurable way. The key insight—that modality preference emerges progressively through layers rather than being static—provides actionable direction for debugging and potentially controlling multimodal model behavior.
How It Works
The approach begins with a conflict-based benchmark that pairs examples where text and visual information contradict each other, requiring the model to exhibit explicit preference through its output. When an OLLM processes such conflicting inputs through its unified representation space, the modality selection rate metric tracks how often the model's response aligns with visual vs. textual ground truth. The authors then perform layer-wise probing by instrumenting the model at different depths to measure when and how modality preference crystallizes during forward propagation. They instrument intermediate representations to determine at which layers the model begins to bias toward visual information, showing that early layers maintain relative balance while mid-to-late layers progressively strengthen visual preference. This progressive emergence suggests the preference emerges from learned attention patterns and cross-modal gating mechanisms rather than input-level biases.
Production Impact
For engineers building multimodal systems, understanding modality preference is crucial for prediction reliability in real-world scenarios with conflicting information—for example, document understanding where OCR text might contradict handwritten annotations or video understanding where narration conflicts with visual content. If your production pipeline assumes balanced modality handling but the deployed OLLM exhibits strong visual preference, your system will silently fail on text-heavy tasks or inconsistent datasets. This work enables concrete debugging: engineers can now measure modality preference in their own models and take corrective action like balanced data sampling, loss weighting during fine-tuning, or prompt engineering to explicitly instruct the model to trust certain modalities. The layer-wise insights suggest that controlling preference at training time (by intervening in mid-to-late layers) may be more efficient than post-hoc correction, reducing inference overhead.
Limitations and When Not to Use This
The paper evaluates only ten representative OLLMs, so the generalization to newer or proprietary model variants is uncertain; the visual preference trend may not hold across different architectures or training objectives. The conflict-based benchmark, while clever, may not reflect real-world modality conflicts proportionally—synthetic conflicts can behave differently from natural distribution shift. The paper doesn't provide a causal mechanism for why visual preference emerges, only when and how much; understanding the root cause (data bias, architectural inductive biases, or training dynamics) requires deeper analysis. Additionally, the work doesn't propose concrete mitigation strategies beyond observing the phenomenon, leaving it to practitioners to determine how to adjust for this behavior in their systems.
Research Context
This work builds on the recent shift from pipeline-based vision-language models (VLMs like CLIP + LLM) to unified multimodal architectures (like GPT-4V, Qwen-VL, LLaVA variants) that represent all modalities in shared embedding spaces. It directly addresses a gap identified in multimodal model analysis: while prior work measured task accuracy, nobody systematically quantified which modality dominates when models must choose. The conflict-based evaluation approach extends prior work on probing language model internals (mechanistic interpretability) into the multimodal domain, showing that layer-wise analysis can reveal preference emergence. This research opens a new direction in multimodal model diagnosis: understanding and controlling modality preference could become a standard evaluation metric alongside accuracy, similar to how fairness and bias audits are now standard in NLP.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::