Information Router for Mitigating Modality Dominance in Vision-Language Models
| Authors | Seulgi Kim et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.16264 |
| Download | |
| Categories | cs.CV, cs.LG |
Abstract
Vision Language models (VLMs) have demonstrated strong performance across a wide range of benchmarks, yet they often suffer from modality dominance, where predictions rely disproportionately on a single modality. Prior approaches primarily address this issue by steering model's attention allocation, implicitly assuming that all modalities provide sufficient information. However, attention only determines where the model focuses, and cannot enrich information that is missing or ambiguous. In the real world, input modalities often differ in information density and their signal-to-noise ratios. In such cases, simply adjusting model's attention does not resolve the underlying lack of information. In this paper, we propose \textsc{MoIR}: \textit{Multi-modal Information Router}, an information-level fusion method that explicitly reduces information disparity prior to fusion. \textsc{MoIR} identifies less informative tokens and routes complementary information from a stronger modality, constructing information-dense token representations before they are processed by a large language model. By modifying information availability, \textsc{MoIR} enables reliable shifts in modality dominance, even when one modality is degraded. We evaluate \textsc{MoIR} on three widely used multi-modal benchmarks across multiple model backbones. Experimental results show that \textsc{MoIR} consistently demonstrates more balanced modality contribution, and improves robustness and downstream performance, particularly even under modality degradation. These findings demonstrate that explicitly modifying cross-modal information is an effective and complementary strategy for mitigating modality dominance in multi-modal reasoning models.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in Vision-Language Models (VLMs) where one modality (vision or language) dominates predictions, causing the model to ignore information from the other modality. Rather than just adjusting where the model pays attention, the authors propose MoIR (Multi-modal Information Router), which actively routes and enriches information from weaker modalities when they contain missing or ambiguous signals. The key insight is that real-world inputs have different information densities and signal-to-noise ratios, so simply redirecting attention doesn't solve the underlying lack of information—you need to actively route and enhance the data itself.
Core Technical Contribution
The core novelty is replacing attention-based steering with an information routing mechanism that operates at the feature level rather than the attention level. MoIR explicitly models information density and signal quality across modalities, then dynamically routes features to compensate when one modality is information-poor or noisy. This is fundamentally different from prior work (which assumes all modalities are informationally complete and just needs attention rebalancing) by recognizing that modality dominance is often a data problem, not an attention problem. The router acts as a gating and enrichment mechanism that can suppress uninformative features and amplify complementary signals from the weaker modality.
How It Works
The system takes multimodal inputs (image and text embeddings from a frozen VLM backbone) and routes them through learned routing gates that assess information quality and complementarity. For each modality, the router computes routing weights based on estimated information density—measuring how much unique, non-redundant signal each modality contributes to the prediction. When one modality is weak or noisy, the router increases the weight of the complementary modality's features and can optionally apply feature transformation to enrich the weaker signal. The routing decisions are learned end-to-end during training, allowing the model to adapt dynamically to different input characteristics rather than applying static attention adjustments.
Production Impact
In production VLM pipelines (e.g., image-text retrieval, visual question answering, captioning), this approach directly improves robustness by preventing catastrophic failure modes where the model ignores entire modalities. For example, in medical imaging reports, when text is ambiguous, the router will amplify visual evidence; conversely, when images are noisy, it routes to text. The computational overhead is minimal—just additional gating layers and routing computations on top of the frozen backbone—making it practical to retrofit into existing systems. Integration is straightforward: you add the routing module between the backbone embeddings and the task head, requiring only fine-tuning on your task-specific data rather than retraining from scratch. Trade-off: requires diverse training data that covers cases where individual modalities are weak, or performance gains may not transfer to production distributions.
Limitations and When Not to Use This
The paper's approach assumes that modality weakness is information-theoretic (missing/noisy data) rather than arising from fundamental misalignment between modalities—cases where modalities are orthogonal or contradictory may not benefit from routing. It requires task-specific fine-tuning on data with varied information densities, which may not exist or be labeled in all domains, limiting zero-shot applicability. The method doesn't address cases where the dominant modality is actually more reliable and should dominate (e.g., in safety-critical tasks, ignoring a warning text to trust a misleading image would be catastrophic). Additionally, the paper doesn't discuss failure modes where routing gates become miscalibrated on distribution shifts—if production data has different modality characteristics than training, the router may make worse decisions than simple attention baselines.
Research Context
This work extends the line of research on multimodal fusion and modality balancing in VLMs, building on prior attention-steering approaches like those in BLIP and CLIP variants. It positions modality dominance as a feature engineering problem rather than purely an attention allocation problem, opening a new research direction: explicit information routing and quality estimation in multimodal systems. The paper likely benchmarks on standard VLM evaluation sets (COCO, Flickr30K, VQA, Image-Text retrieval) and possibly contributes new evaluation protocols that test robustness to information-poor modalities. This work complements concurrent research on modality balancing and could influence future VLM architectures to incorporate adaptive feature routing as a standard component.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::