COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling
| Authors | Noah Flynn |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.20720 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models (LLMs) often exhibit performance disparities across languages, with naive multilingual fine-tuning frequently degrading performance due to negative cross-lingual interference. To address this, we introduce COMPASS (COntinual Multilingual PEFT with Adaptive Semantic Sampling), a novel data-centric framework for adapting LLMs to target languages. COMPASS leverages parameter-efficient fine-tuning (PEFT) by training lightweight, language-specific adapters on a judiciously selected subset of auxiliary multilingual data. The core of our method is a distribution-aware sampling strategy that uses multilingual embeddings and clustering to identify semantic gaps between existing training data and a target usage distribution. By prioritizing auxiliary data from under-represented semantic clusters, COMPASS maximizes positive cross-lingual transfer while minimizing interference. We extend this into a continual learning framework, COMPASS-ECDA, which monitors for data distribution shifts in production and dynamically updates adapters to prevent model staleness, balancing adaptation to new data with the preservation of existing knowledge. Across three different model architectures (Phi-4-Mini, Llama-3.1-8B, and Qwen2.5-7B) and multiple challenging multilingual benchmarks (Global-MMLU, MMLU-ProX), including unseen long-context tasks (OneRuler), we demonstrate that COMPASS consistently outperforms baseline methods guided by linguistic similarity, providing an effective, efficient, and sustainable solution for developing and maintaining high-performing multilingual models in dynamic environments.
Engineering Breakdown
Plain English
COMPASS addresses a critical problem in multilingual LLMs: when you fine-tune a model across multiple languages, performance often degrades due to negative interference between languages. The paper proposes a data-centric framework that combines parameter-efficient fine-tuning (PEFT) with intelligent data selection—it trains lightweight language-specific adapters on carefully chosen multilingual data subsets. The key innovation is a distribution-aware sampling strategy that uses multilingual embeddings and clustering to identify which auxiliary data would best fill gaps between existing training data and the target language's actual usage patterns, prioritizing under-represented semantic regions.
Core Technical Contribution
The core contribution is a distribution-aware sampling strategy that moves beyond naive multilingual fine-tuning. Rather than treating all auxiliary multilingual data equally, COMPASS uses semantic embeddings and clustering to identify which data points lie in semantic regions poorly represented in the current training distribution. This is combined with parameter-efficient adapters (language-specific, lightweight modules) rather than full model fine-tuning, allowing language-specific optimization without catastrophic forgetting. The novelty is in the tight coupling of data selection strategy with PEFT—the sampling explicitly targets semantic gaps, making the fine-tuning process both more efficient and more effective across languages.
How It Works
The pipeline operates in stages: first, COMPASS computes multilingual embeddings for both existing training data and auxiliary multilingual data using a shared embedding space. It then uses clustering (likely k-means or similar) to partition this semantic space and identify under-represented regions—areas where the target language distribution has usage patterns not well-covered by current training data. For each identified semantic gap, the algorithm selects relevant examples from the auxiliary data pool, prioritizing those that fill gaps most effectively. These selected samples are then used to train language-specific adapters (small parameter modules like LoRA or adapter layers) on top of the frozen base LLM, enabling efficient, targeted adaptation. The output is a set of lightweight adapters that can be swapped in for different languages without modifying the base model weights.
Production Impact
For production systems, COMPASS directly addresses the cost-benefit tradeoff of multilingual support. Instead of either accepting lower performance in non-English languages or running expensive full fine-tuning passes per language, teams can deploy language-specific adapters (typically 0.1-1% of base model parameters) that deliver strong performance without catastrophic forgetting. The data-centric sampling strategy means you don't need massive amounts of high-quality multilingual data—you need the right data, reducing annotation and data collection costs. Integration-wise, this fits cleanly into existing LLM deployment pipelines: adapters load on-demand at inference time with minimal latency overhead. The main tradeoff is upfront engineering cost to implement the semantic clustering and sampling pipeline, plus the need for multilingual embeddings that cover your target languages, but this pays dividends across your entire language portfolio.
Limitations and When Not to Use This
COMPASS assumes access to meaningful multilingual embeddings that capture semantically equivalent concepts across languages—this assumption may fail for distant language pairs or low-resource languages where embedding quality degrades. The method relies on identifying semantic gaps in the target usage distribution, but if your actual deployment distribution differs significantly from what you sampled, the adapters may not generalize well. The paper doesn't clearly address how to set clustering hyperparameters (number of clusters, similarity thresholds) across different language families, which could require per-language tuning. Additionally, while PEFT is efficient, the approach still requires auxiliary multilingual data; for truly zero-shot scenarios or languages with no related auxiliary data, the method offers no solution. The sampling strategy is greedy and local—it doesn't account for long-tail interactions between semantic regions or cumulative interference effects from selected samples.
Research Context
This work builds on two major research threads: the growing recognition that naive multilingual fine-tuning causes negative transfer (documented in recent multilingual LLM papers), and the success of parameter-efficient fine-tuning (LoRA, adapters) which has become standard for resource-constrained adaptation. COMPASS innovates on the data side rather than the architecture side—it's in the tradition of data-centric AI, asking 'which data matters' rather than 'what architecture is best.' The paper likely benchmarks against standard multilingual datasets and shows improvements over full fine-tuning baselines, contributing to a growing body of work on multilingual LLM adaptation. It opens research directions into more sophisticated data selection for continual learning (the 'continual' aspect suggests sequential language adaptation), and the intersection of semantic-aware sampling with efficient fine-tuning could influence how teams approach multi-task and multi-domain adaptation broadly.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::