Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers
| Authors | Qingcheng Zeng et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2604.17632 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Code-switching is a pervasive linguistic phenomenon in global communication, yet modern information retrieval systems remain predominantly designed for, and evaluated within, monolingual contexts. To bridge this critical disconnect, we present a holistic study dedicated to code-switching IR. We introduce CSR-L (Code-Switching Retrieval benchmark-Lite), constructing a dataset via human annotation to capture the authentic naturalness of mixed-language queries. Our evaluation across statistical, dense, and late-interaction paradigms reveals that code-switching acts as a fundamental performance bottleneck, degrading the effectiveness of even robust multilingual models. We demonstrate that this failure stems from substantial divergence in the embedding space between pure and code-switched text. Scaling this investigation, we propose CS-MTEB, a comprehensive benchmark covering 11 diverse tasks, where we observe performance declines of up to 27%. Finally, we show that standard multilingual techniques like vocabulary expansion are insufficient to resolve these deficits completely. These findings underscore the fragility of current systems and establish code-switching as a crucial frontier for future IR optimization.
Engineering Breakdown
Plain English
This paper addresses a major gap in information retrieval systems: they work well for single-language queries but fail dramatically when users mix languages in one query (code-switching), which is common in multilingual communities worldwide. The authors created CSR-L, a human-annotated benchmark dataset of code-switched queries, and tested it against state-of-the-art retrieval models using statistical, dense embedding, and late-interaction ranking approaches. Their key finding: code-switching causes significant performance degradation even in multilingual models, and the root cause is a substantial gap in the embedding space representation between pure-language and mixed-language text. This gap acts as a fundamental bottleneck that existing multilingual systems don't handle effectively.
Core Technical Contribution
The core contribution is identifying and quantifying code-switching as a distinct, underexplored failure mode in information retrieval systems, separate from general multilingual challenges. The authors introduce CSR-L, the first human-annotated benchmark specifically designed to measure code-switched IR performance across multiple retrieval paradigms (BM25, dense retrievers, late-interaction rankers). The paper's key insight is that code-switching failures stem from embedding space divergence rather than vocabulary coverage or language understanding—meaning the models have learned separate, non-aligned representations for code-switched versus pure-language text. This diagnostic finding points toward a fundamentally different solution approach than simply improving multilingual training data.
How It Works
The methodology starts with constructing CSR-L through human annotation, capturing authentic code-switched queries from real multilingual users rather than synthetic generation, ensuring naturalness and covering multiple language pairs. Queries are then evaluated against three retrieval paradigms: (1) statistical ranking (BM25-style), (2) dense retrieval using multilingual embeddings (like mBERT or XLM-RoBERTa), and (3) late-interaction methods like ColBERT that score at token granularity. During evaluation, the authors measure retrieval metrics (MRR, nDCG, etc.) and then analyze embedding spaces using techniques like embedding similarity analysis to show where code-switched text representations diverge from monolingual references. The paper traces failure back to the embedding layer by computing cosine similarities between code-switched and monolingual query embeddings, revealing that mixed-language queries produce vectors far from their semantic equivalents in pure language.
Production Impact
For teams building search systems serving multilingual users, this paper exposes a critical blind spot: standard multilingual IR evaluation on monolingual test sets completely misses real-world code-switching performance drops, meaning your production metrics may be lying about actual user experience. If your user base includes code-switchers (immigrant communities, diaspora users, multicultural workplaces), you're likely shipping a retrieval system that underperforms for these users without knowing it. The practical implication is that you need to (1) audit your test sets for code-switched queries, (2) either fine-tune dense retrievers on code-switched data or engineer embedding space alignment techniques, and (3) accept that multilingual models alone won't solve this—you need explicit code-switching adaptation. The cost trade-off: collecting labeled code-switched data is labor-intensive, but the alternative is silently degraded performance for a significant user cohort; compute cost for fine-tuning is modest compared to the performance gains you'd recover.
Limitations and When Not to Use This
The paper focuses on information retrieval and doesn't address code-switching in other NLP tasks (generation, classification, QA), so results may not generalize beyond retrieval. CSR-L is described as 'lite,' suggesting limited scale or language pair coverage—the paper doesn't clarify how many query pairs, language combinations, or domains are included, raising questions about whether findings hold across diverse code-switching patterns (subject-based switching vs. discourse-based switching). The root cause analysis identifies embedding divergence but doesn't propose or validate solutions; the paper diagnoses the problem without demonstrating effective remediation techniques, leaving practitioners without clear next steps beyond data augmentation. The paper assumes code-switching is consistently natural and frequent in user queries, but real-world frequency and user distribution across code-switching behaviors may vary dramatically by region, application domain, and user demographics.
Research Context
This work builds on a long tradition of multilingual NLP research (XLM, mBERT, multilingual T5) but identifies code-switching as a distinct phenomenon requiring dedicated study, similar to how research on domain adaptation separated itself from general transfer learning. It contributes to the information retrieval evaluation literature by introducing a new benchmark (CSR-L) alongside existing multilingual IR benchmarks like mMARCO and mr-TYDI, filling a gap in how retrieval is assessed in multilingual settings. The embedding space analysis connects to theoretical work on representation learning and cross-lingual transfer, questioning assumptions that multilingual embeddings automatically align code-switched text. This opens a research direction on embedding space alignment techniques, potential architectures like language-aware pooling, and fundamental questions about whether code-switching requires fundamentally different model designs versus better training objectives.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::