Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs
| Authors | Jaechul Roh & Amir Houmansadr |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2604.16659 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Prior work shows that fine-tuning aligned models on benign data degrades safety in text and vision modalities, and that proximity to harmful content in representation space predicts which samples cause the most damage. However, existing analyses operate within a single, undifferentiated embedding space -- leaving open whether distinct input properties drive the vulnerability differently. Audio introduces a structurally richer problem: a benign sample can neighbor harmful content not only through what is said but through how it sounds, even when its words are entirely innocuous. We present the first systematic study of benign fine-tuning safety in Audio LLMs, evaluating three state-of-the-art models with a proximity-based filtering framework that selects benign audio by embedding-space distance to harmful content. By decomposing proximity into semantic, acoustic, and mixed axes using external reference encoders alongside each model's own internal encoder, we show that benign fine-tuning elevates Jailbreak Success Rate (JSR) from single digits to as high as 87.12%. Crucially, the dominant vulnerability axis and the relative risk of audio versus text fine-tuning are both architecture-conditioned -- determined by how each model's encoder and projector transform audio into the LLM's input space. We propose two defenses: filtering training data to maximize distance from harmful embeddings, and a textual system prompt at inference, both reducing JSR to near-zero without architectural modification. Our mechanistic analysis on two architectures reveals that fine-tuning selectively suppresses the late-layer refusal circuit while the frozen encoder preserves representations, and that even the suppression pattern is architecture-conditioned, mirroring the behavioral asymmetries across modalities. Safety degradation from benign fine-tuning is a qualitatively distinct risk in Audio LLMs.
Engineering Breakdown
Plain English
This paper investigates how fine-tuning audio language models on benign (safe) data can paradoxically degrade their safety guardrails, extending prior findings from text and vision to the audio domain. The key insight is that audio presents a unique vulnerability: a sample with innocent words can still be close in embedding space to harmful content if its acoustic properties (tone, sound, prosody) resemble harmful audio. The authors propose a proximity-based filtering framework that selects benign audio samples based on embedding-space distance, and evaluate it on three state-of-the-art audio LLMs to understand which properties—semantic content versus acoustic characteristics—drive safety degradation.
Core Technical Contribution
The core novelty is the first systematic analysis of benign fine-tuning safety specifically for audio LLMs, recognizing that audio embeddings encode both linguistic and acoustic information, creating a richer attack surface than text or vision alone. Prior work assumed a single, undifferentiated embedding space, but this paper discovers that harmful audio neighbors can exist through multiple independent pathways: word content and acoustic similarity. The authors introduce a proximity-based filtering mechanism that can discriminate which benign samples pose the highest safety risk by measuring their distance to harmful content in the joint audio embedding space, allowing practitioners to avoid the most problematic fine-tuning data.
How It Works
The system operates on fine-tuned audio LLMs by first embedding both benign fine-tuning data and a reference set of harmful audio into a shared embedding space (extracted from an aligned model). For each benign sample, the framework computes its distance to the nearest harmful content in this space using a distance metric (likely cosine or Euclidean). Samples with smaller distances to harmful content are flagged as high-risk for safety degradation. The filtering framework then selectively excludes or down-weights these proximal samples during fine-tuning, reducing the model's exposure to embedding-space neighbors of harmful content. The authors evaluate three state-of-the-art audio LLMs to measure how filtering at different proximity thresholds affects both safety metrics (adversarial robustness, refusal rates) and utility metrics (downstream task performance), establishing which acoustic and semantic properties correlate most strongly with safety degradation.
Production Impact
For teams fine-tuning audio LLMs in production, this work provides a concrete screening tool to identify and exclude high-risk benign training data before deployment, reducing safety incidents without requiring architectural changes. The proximity-based filtering can be implemented as a preprocessing step that runs once during data curation, making it cheap to integrate into existing pipelines with minimal latency impact. However, the approach requires maintaining a reference set of known harmful audio and computing embeddings for all candidate fine-tuning data, which adds data annotation and embedding computation overhead proportional to dataset size. Teams deploying audio LLMs for safety-critical applications (medical transcription, content moderation assistance, financial advising) should adopt this filtering to reduce the risk of alignment degradation, though they must balance the exclusion of benign data against the utility loss from removing valid training examples.
Limitations and When Not to Use This
The paper assumes that proximity in embedding space causally predicts safety degradation, but this relationship may not hold uniformly across all model architectures, training procedures, or audio domains—generalization beyond the three tested models is unclear. The framework requires access to labeled harmful audio examples, which may be expensive to obtain or domain-specific, limiting applicability to systems without such reference data. The paper does not deeply analyze the relative importance of acoustic versus semantic properties in driving safety breakdowns, leaving open whether targeted filtering on one dimension (e.g., acoustic similarity alone) could suffice without removing semantically useful benign data. Finally, the work does not address potential adversarial circumvention: a malicious actor could craft benign-sounding audio with imperceptibly modified acoustic features to evade the proximity filter.
Research Context
This work directly extends recent findings that benign fine-tuning degrades safety in text and vision LLMs (e.g., via representation-space proximity), adapting that insight to audio where the embedding space is structurally richer due to acoustic and linguistic components. It builds on the broader safety alignment literature (RLHF, constitutional AI, gradient-based attacks) by investigating how fine-tuning—a standard post-deployment operation—can unintentionally break carefully aligned models. The research direction opened is the study of multi-modal safety properties: since audio embeds semantic and acoustic information jointly, future work can investigate whether decomposing embeddings (e.g., via disentanglement techniques) enables more surgical safety interventions. This paper also motivates deeper investigation into why benign samples near harmful content in embedding space pose a vulnerability, which could inform better alignment techniques for multimodal LLMs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::