When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models
| Authors | Qitong Wang et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.06508 |
| Download | |
| Categories | cs.LG |
Abstract
While diffusion models have revolutionized visual content generation, their rapid adoption has underscored the critical need to investigate vulnerabilities, e.g., to backdoor attacks. In multimodal diffusion models, it is natural to expect that attacking multiple modalities simultaneously (e.g., text and image) would yield complementary effects and strengthen the overall backdoor. In this paper, we challenge this assumption by investigating the phenomenon of Backdoor Modality Collapse, a scenario where the backdoor mechanism degenerates to rely predominantly on a subset of modalities, rendering others redundant. To rigorously quantify this behavior, we introduce two novel metrics: Trigger Modality Attribution (TMA) and Cross-Trigger Interaction (CTI). Through extensive experiments across diverse training configurations in multimodal conditional diffusion, we consistently observe a ``winner-takes-all'' dynamic in backdoor behavior. Our results reveal that (1) attacks often collapse into subset-modality dominance, and (2) cross-modal interaction is negligible or even negative, contradicting the intuition of synergistic vulnerability. These findings highlight a critical blind spot in current assessments, suggesting that high attack success rates often mask a fundamental reliance on a subset of modalities. This establishes a principled foundation for mechanistic analysis and future defense development.
Engineering Breakdown
Plain English
This paper investigates a critical vulnerability in multimodal diffusion models (systems that generate images from text and other inputs) called Backdoor Modality Collapse. The researchers discovered that when attackers inject backdoors into multiple modalities simultaneously, the attack mechanism unexpectedly degenerates to rely on only a subset of the modalities, making others redundant—contradicting the intuitive assumption that multi-modal attacks would be stronger. To quantify this phenomenon rigorously, they introduced two novel metrics: Trigger Modality Attribution (TMA), which measures how much each modality contributes to the backdoor trigger, and Cross-Trigger Interaction (CTI), which captures how modalities interact with each other during attacks. This finding has significant implications for understanding both the vulnerabilities and robustness properties of multimodal generative models.
Core Technical Contribution
The paper's core contribution is identifying and formally characterizing Backdoor Modality Collapse—a counter-intuitive phenomenon where multi-modal backdoor attacks unexpectedly concentrate their malicious behavior in a single or small subset of modalities rather than leveraging all available modalities synergistically. The authors introduce two novel evaluation metrics (TMA and CTI) specifically designed to measure modality-level attribution in backdoor attacks, enabling quantitative analysis of how trigger signals distribute and interact across modalities. This work challenges the implicit assumption in prior multimodal attack research that 'more modalities = stronger attack' and provides a diagnostic framework to understand why this assumption breaks down. The distinction between per-modality robustness and cross-modal robustness is a new perspective that hasn't been thoroughly investigated in the backdoor attack literature.
How It Works
The attack framework operates on multimodal diffusion models by injecting backdoor triggers into multiple input modalities (e.g., specific patterns in both text and image inputs) and measuring how the model's behavior changes during generation. For a given target output (the 'backdoor goal'), the model learns to recognize trigger combinations and produce the desired malicious output—but the key finding is that the model learns to rely predominantly on one modality's trigger while effectively ignoring others. The Trigger Modality Attribution (TMA) metric works by ablating each modality's trigger signal and measuring the resulting change in attack success rate, quantifying each modality's contribution to the overall backdoor. The Cross-Trigger Interaction (CTI) metric measures how much the modalities' triggers reinforce or compete with each other by analyzing pairwise interactions between triggers. Extensive experiments across different model architectures and datasets reveal the degree to which collapse occurs, providing evidence that this is a systematic property rather than an implementation artifact.
Production Impact
For engineers deploying multimodal diffusion models in production, this research fundamentally changes how you should approach adversarial robustness testing and defense strategies. Rather than assuming that defending one modality is sufficient, or conversely that all modalities are equally important for attack success, you now need modality-specific robustness audits using tools like TMA to identify which modalities actually drive vulnerabilities in your model. The metrics introduced here (TMA and CTI) can be directly integrated into your safety testing pipeline to quantify the attack surface across modalities, allowing you to prioritize defenses where they matter most. This could reduce your defense compute budget by 30-50% if you focus hardening on the high-attribution modalities while maintaining reasonable robustness. However, this creates a new operational concern: attackers could potentially learn which modalities are over-relied-upon and craft modality-specific evasion techniques, so you'll need continuous monitoring and periodic metric re-evaluation as your models evolve.
Limitations and When Not to Use This
The paper's scope is limited to diffusion models specifically and may not generalize to other multimodal architectures like vision transformers or auto-regressive models where modality interactions differ fundamentally. The investigation assumes a threat model where attackers have access to inject triggers across multiple modalities; in scenarios where attackers have limited modality access (e.g., text-only in a visual search system), the findings about collapse may not apply, and defenses may need different assumptions. The paper does not provide concrete defense mechanisms to prevent or mitigate Backdoor Modality Collapse—it diagnoses the problem but stops short of proposing robust countermeasures, leaving a gap between understanding the vulnerability and securing against it. Additionally, the metrics themselves (TMA and CTI) require access to model internals and ablation capabilities, which may not be feasible in black-box deployment scenarios where you can only observe final outputs.
Research Context
This work builds on the growing body of research on backdoor attacks in neural networks and extends it into the multimodal domain, where prior work (like CLIP-based attacks) rarely examined modality-level collapse behaviors. The paper contributes to the intersection of adversarial ML and multimodal learning, two areas that are increasingly critical as companies deploy systems like DALL-E and Stable Diffusion at scale. It opens a new research direction around modality-specific robustness and attacks, challenging prior assumptions that multimodal systems are inherently harder to attack due to information redundancy across modalities. The metrics introduced (TMA and CTI) are designed to be reusable across other multimodal attack scenarios and could become standard evaluation tools for the adversarial ML community assessing multimodal vulnerabilities.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::