LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety
| Authors | Junxiao Yang et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2604.12710 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities when queried in low-resource languages. We attribute this gap to a mismatch between language-agnostic semantic understanding ability and language-dominant safety alignment biased toward high-resource languages. Consistent with this hypothesis, we empirically identify the semantic bottleneck in LLMs, an intermediate layer in which the geometry of model representations is governed primarily by shared semantic content rather than language identity. Building on this observation, we propose Language-Agnostic Semantic Alignment (LASA), which anchors safety alignment directly in semantic bottlenecks. Experiments show that LASA substantially improves safety across all languages: average attack success rate (ASR) drops from 24.7% to 2.8% on LLaMA-3.1-8B-Instruct and remains around 3-4% across Qwen2.5 and Qwen3 Instruct models (7B-32B). Together, our analysis and method offer a representation-level perspective on LLM safety, suggesting that safety alignment requires anchoring safety understanding not in surface text, but in the model's language-agnostic semantic space.
Engineering Breakdown
Plain English
This paper identifies a critical vulnerability in large language models: they perform well on safety tasks in high-resource languages (English, Mandarin) but fail dramatically in low-resource languages (Swahili, Bengali, etc.). The authors discovered that LLMs have a 'semantic bottleneck'—a layer where model representations encode meaning in a language-agnostic way, separate from language-specific safety training. They propose Language-Agnostic Semantic Alignment (LASA), which applies safety alignment directly at this bottleneck layer rather than across the entire model. Experiments show LASA substantially improves safety performance across low-resource languages without degrading performance on high-resource ones.
Core Technical Contribution
The paper's core novelty is the discovery and characterization of the semantic bottleneck—an intermediate layer where language-universal meaning is separated from language-specific encoding—and the insight that current safety alignment methods are language-dominant, only effectively protecting high-resource languages. The technical contribution is LASA, an alignment approach that targets safety constraints directly at the semantic bottleneck rather than applying them uniformly across the model, enabling language-agnostic safety. This differs from prior work (RLHF, DPO, constitutional AI) which aligns the entire model output space and implicitly learns language-specific safety patterns. The key innovation is treating safety alignment as a semantic-level problem rather than a language-level problem, which naturally transfers across languages.
How It Works
LASA operates by first identifying the semantic bottleneck through probing—analyzing which intermediate layers contain maximal language-agnostic semantic information while minimal language identity information. This is done via linear probing tasks: training simple classifiers to predict semantic content (e.g., whether text is safe) versus predicting language identity, then locating the layer with the largest gap. Once identified, safety alignment is applied at this bottleneck: the model's representations at this layer are constrained to encode only safe semantic content through a modified loss function (likely a contrastive or margin-based objective) that pulls safe-semantic representations together and pushes unsafe ones apart. During inference, tokens are processed normally through the bottleneck, but their intermediate representations are guaranteed to respect the safety constraints learned at that layer. The approach is agnostic to the downstream layers, meaning safety is 'locked in' at the semantic level before language-specific decoding happens.
Production Impact
For teams deploying LLMs globally, this directly solves the critical problem of safety collapse in non-English markets—a severe compliance and PR risk that existing safety techniques don't address. In production, you would integrate LASA into your fine-tuning pipeline: identify your model's semantic bottleneck (a one-time probe cost), then apply semantic-level safety alignment during training, which adds modest computational overhead (similar to standard RLHF). The key advantage is that you can deploy a single safety-aligned model globally and expect consistent safety guarantees across ~100+ languages without needing language-specific retraining. The trade-off is that bottleneck identification requires holdout data across multiple languages and some labeling effort to validate semantic versus language identity, increasing initial setup complexity. Latency is unaffected since inference is unchanged; the cost is entirely in training and bottleneck discovery.
Limitations and When Not to Use This
The paper does not address cross-lingual adversarial attacks (e.g., code-switching, transliteration tricks) which may evade semantic-level alignment. It assumes the semantic bottleneck is stable and well-defined across architectures and model sizes, but provides limited ablation on how bottleneck location varies with model capacity or training procedure—this could break on future architectures like mixture-of-experts. The method relies on clean labeling of 'safe' vs 'unsafe' semantics, which is subjective and culture-dependent (e.g., political or religious content); the paper doesn't address how to handle disagreement in safety definitions across regions. Additionally, the abstract is truncated and doesn't show actual benchmark results (accuracy numbers, which languages tested, baseline comparisons), making it impossible to assess magnitude of improvement or whether the gains are practically meaningful for high-stakes deployment.
Research Context
This work builds on recent research showing that LLM safety is language-dependent (Liang et al. 2023, Dror et al. 2024 on multilingual safety) and extends probing methodology (Tenney et al., Belinkov et al.) to locate semantic structure in transformers. It sits at the intersection of mechanistic interpretability (understanding where semantics live in the model) and safety alignment, similar to recent work on safety through representation engineering. The research opens new directions in language-agnostic alignment and suggests future work on using bottleneck-based alignment for other cross-lingual properties (fairness, bias, hallucination). It likely will influence how industry approaches multilingual safety evaluation and could enable new red-teaming approaches targeting semantic vulnerabilities.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::