A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning
| Authors | Tianle Chen & Deepti Ghadiyaram |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.03995 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
As audio-visual multi-modal large language models (MLLMs) are increasingly deployed in safety-critical applications, understanding their vulnerabilities is crucial. To this end, we introduce Multi-Modal Typography, a systematic study examining how typographic attacks across multiple modalities adversely influence MLLMs. While prior work focuses narrowly on unimodal attacks, we expose the cross-modal fragility of MLLMs. We analyze the interactions between audio, visual, and text perturbations and reveal that coordinated multi-modal attack creates a significantly more potent threat than single-modality attacks (attack success rate = 83.43% vs 34.93%).Our findings across multiple frontier MLLMs, tasks, and common-sense reasoning and content moderation benchmarks establishes multi-modal typography as a critical and underexplored attack strategy in multi-modal reasoning. Code and data will be publicly available.
Engineering Breakdown
Plain English
This paper systematically investigates how coordinated attacks across multiple modalities—audio, visual, and text—can fool audio-visual multimodal large language models (MLLMs) in safety-critical applications. The researchers discovered that when these typographic perturbations are combined across modalities, they become far more effective than single-modality attacks, achieving an 83.43% success rate compared to 34.93% for unimodal attacks. The work evaluates this vulnerability across multiple frontier MLLMs on common-sense reasoning and content moderation tasks, exposing a critical cross-modal fragility that prior research had not comprehensively studied.
Core Technical Contribution
The core novelty is the first systematic characterization of cross-modal attack coordination in audio-visual MLLMs, moving beyond prior work that examined only unimodal perturbations. The authors introduce a methodology for analyzing how typographic attacks in one modality interact with and amplify perturbations in other modalities, revealing that the threat landscape for MLLMs is fundamentally different when considering multi-modal coordination. This represents a shift from treating attack surfaces as isolated per-modality problems to understanding them as interconnected vulnerabilities where alignment failures across modalities create disproportionate failure modes. The 2.4x success rate improvement (83.43% vs 34.93%) quantifies the practical severity of this cross-modal interaction effect.
How It Works
The attack methodology operates by introducing coordinated typographic perturbations—intentional misspellings, symbol substitutions, and visual distortions—simultaneously across the audio transcription, visual elements, and text components of the input. For audio, the system likely corrupts the transcribed text representation; for vision, it applies visual degradation or character-level perturbations to on-screen text; for text, it directly modifies input tokens. The MLLM must then process all three corrupted modalities together, which requires consistent alignment across its fusion layers to produce correct outputs. When perturbations are isolated to one modality, the model can rely on the uncorrupted information from other modalities to correct the error; however, when all modalities carry coordinated corruptions, the model's cross-modal reconciliation mechanisms fail catastrophically. The study measures success rate—the percentage of attacks that cause incorrect outputs—as the primary evaluation metric across different perturbation intensities and modality combinations.
Production Impact
For engineers deploying audio-visual MLLMs in safety-critical domains (autonomous vehicles, medical diagnosis, accessibility systems), this research creates an urgent validation requirement: models must be tested not just for robustness to unimodal noise, but for vulnerability to coordinated cross-modal attacks. A production pipeline would need to implement adversarial robustness testing that generates multi-modal perturbations during evaluation, adding computational overhead to testing phases. This suggests that filtering or cleaning inputs should operate at the fusion layer rather than per-modality, and that inference services should implement confidence thresholds that account for multi-modal alignment—if all modalities disagree, the system should abstain rather than guess. Teams using frontier MLLMs for high-stakes decisions would need to significantly increase their test coverage to include adversarial attack scenarios, extending development timelines and requiring specialized security expertise.
Limitations and When Not to Use This
The paper does not address defenses or mitigation strategies—it identifies the vulnerability but does not propose solutions beyond the implicit suggestion that multi-modal robustness training is needed. The evaluation is constrained to typographic attacks specifically; other perturbation types (adversarial noise, semantic attacks, jailbreaks) may exhibit different cross-modal interaction patterns that this work doesn't explore. The study is limited to current-generation frontier MLLMs, and it's unclear whether architecture improvements (better fusion mechanisms, attention patterns, or training objectives) will reduce susceptibility to these attacks. Additionally, the paper doesn't distinguish between different types of coordinated attacks (synchronized vs. asynchronous perturbations, partial vs. full corruption) or establish how much coordination is actually necessary to achieve high success rates.
Research Context
This work extends the adversarial robustness literature from unimodal models (where audio and vision security have been studied separately) to the multi-modal setting that increasingly dominates frontier LLMs. It builds on prior research in adversarial examples and robustness but identifies a previously underexplored dimension: the interaction surface between modalities as a distinct attack vector. The paper contributes to the growing body of work on MLLM safety and alignment, alongside research on jailbreaks, hallucinations, and prompt injection, establishing multi-modal coordination as a critical evaluation dimension for safety benchmarks. This opens a research direction for developing multi-modal robustness training methods, fusion-layer defenses, and evaluation frameworks that treat cross-modal alignment as a security property rather than just a capability property.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::