LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
| Authors | Jianan Chen et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.20042 |
| Download | |
| Categories | cs.CL, cs.AI |
Abstract
Large language models (LLMs) have driven substantial advances in speech language models (SpeechLMs), yielding strong performance in automatic speech recognition (ASR) under high-resource conditions. However, existing benchmarks predominantly focus on high-resource languages, leaving the ASR behavior of SpeechLMs in low-resource languages insufficiently understood. This gap is critical, as practical ASR systems must reliably support low-resource languages and generalize across diverse language families, and it directly hinders the deployment of SpeechLM-based ASR in real-world multilingual scenarios. As a result, it is essential to evaluate SpeechLMs on low-resource languages to ensure their generalizability across different language families. To address this problem, we propose \textbf{LoASR-Bench}, a comprehensive benchmark designed to evaluate \textbf{lo}w-resource \textbf{a}utomatic \textbf{s}peech \textbf{r}ecognition (\textbf{ASR}) of the latest SpeechLMs across diverse language families. LoASR-Bench comprises 25 languages from 9 language families, featuring both Latin and non-Latin scripts, enabling cross-linguistic and cross-script assessment of ASR performance of current SpeechLMs. Experimental results highlight the limitations of the latest SpeechLMs in handling real-world low-resource languages.
Engineering Breakdown
Plain English
This paper addresses a critical gap in speech language model (SpeechLM) evaluation: existing benchmarks focus almost entirely on high-resource languages, leaving their performance on low-resource languages largely unknown. The authors identify that while LLMs have driven substantial advances in ASR for well-resourced conditions, deploying these systems in real-world multilingual scenarios requires understanding how they generalize across diverse language families with limited training data. The paper proposes a comprehensive evaluation framework (which appears to be cut off in the abstract) to systematically assess SpeechLM behavior on low-resource languages, ensuring that ASR systems built on these models can reliably support global language coverage.
Core Technical Contribution
The core contribution is establishing a rigorous benchmark and evaluation methodology for measuring SpeechLM performance on low-resource languages—a dimension that has been systematically neglected despite its practical importance. Rather than proposing a new model architecture, the authors contribute a diagnostic framework that reveals failure modes and generalization gaps when SpeechLMs encounter languages with limited training data and diverse phonological/grammatical structures. This work shifts the evaluation paradigm from optimizing for high-resource scenarios to stress-testing robustness across the long tail of global languages, which is essential for fair and practical ASR system deployment.
How It Works
The evaluation framework likely works by constructing a diverse benchmark of low-resource language speech datasets spanning multiple language families (e.g., Sino-Tibetan, Afro-Asiatic, Austronesian, etc.), ensuring coverage of varying degrees of data scarcity. For each language, the authors would apply existing SpeechLM architectures (built on top of LLM backbones fine-tuned or adapted for speech) and measure standard ASR metrics like Character Error Rate (CER) and Word Error Rate (WER). The pipeline involves: (1) selecting or creating low-resource speech corpora with proper linguistic annotation, (2) adapting pre-trained SpeechLMs to these languages with minimal fine-tuning data, (3) establishing baseline performance, and (4) analyzing failure patterns to identify which linguistic features, acoustic conditions, or language-family characteristics correlate with degraded performance. The output is a detailed performance matrix showing how generalization varies with data scarcity, language typology, and model capacity.
Production Impact
For engineers deploying multilingual ASR systems, this work directly informs which SpeechLM architectures and fine-tuning strategies are safe to use for under-resourced languages where you cannot afford performance failures. Production systems serving global users (e.g., translation platforms, accessibility tools, voice interfaces in emerging markets) would use these benchmarks to make go/no-go decisions: is a given SpeechLM variant reliable enough for Amharic, Bengali, or Swahili? The practical impact is reducing risk of deploying models that silently degrade on populations speaking low-resource languages. Trade-offs include the need to maintain language-specific evaluation datasets (creating annotation overhead), potential requirements for per-language fine-tuning rather than pure zero-shot transfer, and the possibility that some language families simply lack sufficient public data, necessitating data collection investments or ensemble approaches with fall-back systems.
Limitations and When Not to Use This
The paper does not propose solutions to improve SpeechLM performance on low-resource languages—it diagnoses the problem rather than solving it. The evaluation likely assumes access to reasonable test sets for low-resource languages, which may not exist for the world's most critically endangered languages, limiting the coverage of the benchmark. The framework probably does not address code-switching (a real-world phenomenon in multilingual communities) or highly noisy audio conditions (common in low-resource regions), which would further constrain applicability. Additionally, the paper's impact depends on the speech SpeechLM community adopting the benchmark; if existing models are not systematically re-evaluated on this benchmark, the work risks remaining a diagnostic tool that guides future research without immediately shifting industry practice.
Research Context
This work builds on recent advances in SpeechLMs (like wav2vec, HuBERT, and their LLM integration variants) that have achieved state-of-the-art results in high-resource ASR but have not been systematically stress-tested on language diversity. It extends evaluation practices from the NLP community (which has increasingly scrutinized model performance on underrepresented languages) into the speech domain. The paper opens a research direction around fairness and equitable AI—ensuring that speech technology does not inadvertently create a digital divide where only speakers of well-resourced languages benefit from LLM-powered ASR advances. This work will likely motivate follow-up research in data augmentation for low-resource speech, transfer learning strategies that preserve cross-lingual generalization, and multilingual pre-training objectives designed to balance performance across language families rather than optimizing for average-case high-resource languages.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::