Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs
| Authors | Jayadev Billa |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23136 |
| Download | |
| Categories | cs.CL, cs.AI, cs.LG |
Abstract
Multimodal LLMs can process speech and images, but they cannot hear a speaker's voice or see an object's texture. We show this is not a failure of encoding: speaker identity, emotion, and visual attributes survive through every LLM layer (3--55 above chance in linear probes), yet removing 64--71% of modality-specific variance improves decoder loss. The decoder has no learned use for these directions; their presence is noise. We formalize this as a mismatched decoder problem: a decoder trained on text can only extract information along text-aligned directions. Accessible information is bounded by the Generalized Mutual Information (GMI), with degradation scaling with distributional distance and decoder sensitivity. The bound is a property of the decoder's scoring rule, not of any particular architecture; it applies whether non-text inputs arrive through a learned projection, a discrete codebook, or no explicit adapter at all. We validate this across five models spanning speech and vision. A controlled experiment (two Prismatic VLMs differing only in encoder text-alignment) confirms the bottleneck is the decoder's scoring rule, not the encoder or projection. A LoRA intervention demonstrates the fix: training with an emotion objective improves emotion accessibility (7.5%) without affecting other attributes, confirming that the training objective determines what becomes accessible.
Engineering Breakdown
Plain English
This paper investigates why multimodal large language models fail to preserve speaker identity, emotion, and visual texture information even though these details survive through all transformer layers (3-55x above chance). The authors discover the real problem isn't in encoding—it's in the decoder: removing 64-71% of modality-specific variance actually improves decoder loss, meaning the decoder has learned to ignore these signals as noise. They formalize this as a 'mismatched decoder' problem where a text-trained decoder can only extract information along directions aligned with text, and prove an information-theoretic bound (Generalized Mutual Information) that predicts how much accessible information degrades based on distributional distance between modalities.
Core Technical Contribution
The paper's core insight is reframing multimodal collapse not as an encoding failure but as a fundamental decoder mismatch problem. The authors prove that a decoder trained on text-only objectives cannot access modality-specific information even when that information is perfectly preserved in intermediate representations—this is an information-theoretic limitation, not a capacity issue. They introduce the Generalized Mutual Information (GMI) bound to quantify accessible information across modalities, showing degradation scales predictably with distributional distance and decoder sensitivity. This shifts the lens from 'why aren't we encoding enough' to 'the decoder's scoring function was never optimized to use these directions,' suggesting solutions must change training objectives rather than architecture alone.
How It Works
The paper starts by probing multimodal LLM representations at each layer using linear classifiers to measure how much speaker identity, emotion, and visual attributes are preserved (baseline is random chance). They then perform spectral analysis to identify and remove the top 64-71% of modality-specific variance from hidden states, measuring the impact on decoder loss. The key finding: removing this variance improves loss, indicating the decoder treats these directions as noise. Mathematically, they formalize this using Generalized Mutual Information (GMI), which bounds the information accessible to a decoder based on the decoder's scoring rule (not model weights). The GMI bound depends on distributional distance between the speech/image modality and text, scaled by the decoder's sensitivity; when distributions are far apart, even well-preserved representations yield low accessible information because the decoder's objective was optimized only for text alignment.
Production Impact
For production systems, this paper explains why expensive multimodal encoders don't yield expected gains in speaker recognition, emotion detection, or texture-aware vision tasks—the bottleneck is the text-aligned decoder, not representation quality. Engineers can now diagnose this systematically: if linear probes show modality-specific information is preserved but end-task performance is poor, the issue is decoder mismatch, not encoder failure. Solutions include retraining decoders with joint objectives (e.g., contrastive losses that reward modality-specific directions), or using separate task-specific decoders instead of a single text-aligned head. The trade-off is non-trivial: multi-objective training adds complexity and may hurt text performance slightly, while separate decoders increase model size and inference cost. The paper suggests that for applications requiring speaker or texture information (voice biometrics, accessibility features), the current architecture may be fundamentally limited without decoder redesign.
Limitations and When Not to Use This
The paper assumes modality-specific information is recoverable via linear probing, but doesn't address whether nonlinear classifiers might access these directions more effectively—if they do, the 'decoder mismatch' may be less severe than claimed. The GMI bound is derived for a specific class of scoring rules and may not tightly characterize all modern decoder designs (e.g., ensemble decoders, adaptive routing mechanisms). The analysis focuses on the LLM decoder in isolation and doesn't examine end-to-end fine-tuning scenarios where decoder and encoder co-adapt, which could partially mitigate the mismatch. Finally, the paper doesn't provide concrete solutions or experiments validating whether multi-objective training or decoder redesign actually recovers the 64-71% of lost modality-specific information in downstream tasks—that gap between theory and practice remains open.
Research Context
This work builds on a growing body of literature showing that multimodal models suffer from modality imbalance and that text-only training dominates learning dynamics (related to modality dumbing-down in vision-language models). It advances prior work on linear probing interpretability by using spectral analysis to directly measure the cost of removing learned representations. The paper contributes to the theoretical understanding of information bottlenecks in multimodal systems, complementing empirical work on modality dropout and scaling laws. Future research could extend this framework to other modality pairs (audio-vision, language-action), explore whether task-specific decoders recover performance, and develop practical training recipes that optimize for modality preservation while maintaining text alignment.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::