Anisotropic Modality Align
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 11 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Xiaomin Yu et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2605.07825 |
| Download | |
| Code | https://github.com/Yu-xm/Modality_Gap_Theory |
Abstract
Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data. Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data. However, the key premise of this paradigm remains insufficiently understood: can representations from different modalities be reliably interchanged? The core obstacle lies in the persistent Modality Gap in the shared space. In this work, we revisit the geometric nature of the modality gap. We find that modality representations already share compatible dominant semantic geometry. What truly hinders modality interchangeability is not a simple global shift, but an anisotropic residual structure concentrated along a small number of dominant directions. Based on this finding, we further propose the principle of anisotropic modality gap alignment: effective modality alignment should align with the target-modality distribution while preserving the semantic structure of the source modality. Guided by this principle, we propose an anisotropic geometric correction framework, AnisoAlign, for unpaired modality alignment. This framework leverages the internal geometric prior of the target modality and performs bounded correction on source-modality representations, thereby constructing substitute representations in the target modality. Experiments confirm its benefits in both geometric diagnostics and text-only MLLM training. Overall, this work recasts the modality gap from an empirical observation into a correctable, structured geometric phenomenon and provides a new representation alignment perspective for training multimodal models with unimodal data.
Engineering Breakdown
Plain English
This paper investigates why multimodal language models struggle to reliably use representations from different modalities (text, images, etc.) interchangeably, even when trained on shared representation spaces. The researchers discovered that the core problem isn't a simple global mismatch between modalities—the dominant semantic geometry is actually compatible—but rather an anisotropic (directionally-biased) residual structure that prevents straightforward interchangeability.
Key Engineering Insight
Modality gap isn't a uniform distance problem you can fix with a global transformation. The incompatibility is anisotropic, meaning it varies by direction in the representation space, which means naive alignment strategies will fail. You need directionally-aware alignment, not just centering or scaling.
Why It Matters for Engineers
Training multimodal LLMs with only unimodal data (text-only or image-only) would massively reduce data requirements and unlock cheaper, faster model training. Right now this doesn't work reliably because you can't safely swap representations between modalities. Understanding and fixing the anisotropic gap directly unblocks this practical scaling path for production systems.
Research Context
Prior work showed that pretrained contrastive models create usable shared spaces where unimodal data could theoretically train multimodal models, but this only worked inconsistently. This paper reframes the problem: the gap isn't a global alignment issue but a geometric one rooted in anisotropic residuals. This shifts the solution space from simple alignment techniques to geometry-aware methods, enabling more reliable multimodal training with scarce paired data.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::