Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
| Authors | Yuchen Zhang et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.06505 |
| Download | |
| Categories | cs.CL |
Abstract
Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to monolingual settings and short, isolated utterances. While recent efforts in context-aware ASR show promise, two key challenges persist: limited multilingual support and the absence of principled alignment between speech and contextual representations. In this paper, we introduce a context-aware multilingual ASR framework that supports diverse languages and accents while preserving the modularity of pretrained models. Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module, allowing structured context prompts, including dialogue history and biasing words, to guide transcription. To improve interaction between speech and context, we employ a contrastive learning objective that aligns their representations in a shared embedding space. Evaluations on over 1,500 hours of real-world conversational speech across 11 languages and 5 English dialects show that contextual input consistently improves recognition quality. Contrastive alignment provides additional gains when applied to different context types, with an overall performance gain of over 5%. These results highlight the importance of both contextual modeling and cross-modal alignment in multilingual ASR.
Engineering Breakdown
Plain English
This paper addresses a critical gap in automatic speech recognition (ASR) systems: most current models work only in single languages and cannot leverage contextual information like dialogue history or domain-specific biasing words. The authors propose a context-aware multilingual ASR framework that freezes a pretrained speech encoder and language model, connecting them via a lightweight projection module that accepts structured context prompts to improve transcription accuracy. The approach preserves modularity—you can swap in different pretrained models without retraining—while supporting multiple languages and accents. The key innovation is a principled alignment mechanism between speech representations and contextual representations that guides what the model transcribes.
Core Technical Contribution
The core technical novelty is a modular projection-based alignment layer that bridges frozen pretrained speech encoders and decoder-only language models without full retraining. Unlike prior context-aware ASR work that tightly couples components or requires monolingual training, this framework uses structured context prompts (dialogue history, speaker information, domain biasing words) as explicit inputs to the decoder, allowing the projection module to learn how to blend speech acoustics with contextual semantics. The approach is multilingual by design—it doesn't require language-specific fine-tuning—because both the speech encoder and language model are already pretrained on diverse data. The principled alignment differs from prior ad-hoc fusion methods by treating context injection as a learned projection problem rather than concatenation or gating.
How It Works
The system takes raw speech audio as input to a frozen pretrained speech encoder (likely a model like Wav2Vec2 or HuBERT), producing a sequence of acoustic embeddings. In parallel, structured context is formatted as discrete tokens (e.g., 'DIALOGUE_HISTORY: [previous utterance] BIAS_WORDS: [domain terms]') and fed into a frozen decoder-only language model (similar to GPT). A lightweight projection module—trainable during fine-tuning—learns a bilinear or cross-attention transformation that fuses the acoustic embeddings with contextual embeddings into a shared representation space. This fused sequence is then passed to the language model's decoder head (or to a separate transcription decoder) to generate the final transcript tokens. The projection module is the only component trained on labeled ASR data, keeping compute costs low and enabling rapid adaptation to new languages by simply swapping frozen encoders.
Production Impact
In production, this approach solves two concrete problems: (1) handling user context without retraining the entire model—you can dynamically inject dialogue history and domain keywords at inference time, which dramatically improves accuracy for domain-specific tasks like medical or legal transcription; (2) supporting multilingual/multilingual pipelines with a single architecture, eliminating the need to maintain separate ASR models per language. The main trade-off is inference latency: the projection module adds a forward pass, and passing large context sequences (long dialogue history) increases computation at decode time. Memory footprint is favorable because you only store and load the lightweight projection—the frozen encoders and language models are already deployed. Integration complexity is moderate: you need to standardize how context is tokenized and formatted, and design an interface for dynamically injecting context at inference time (e.g., via a JSON payload). Real-world challenge: the quality of your transcription now depends heavily on context quality and relevance—garbage context tokens hurt more than no context.
Limitations and When Not to Use This
The paper's abstract does not disclose key limitations, but several are predictable: (1) the approach assumes both the speech encoder and language model are pretrained on high-quality multilingual data—if you have a low-resource language without good pretrained models, you lose the modularity advantage; (2) structured context injection requires careful prompt engineering, and the system likely has failure modes when context is malformed, contradictory, or includes out-of-vocabulary terms; (3) the lightweight projection assumes the acoustic and semantic embedding spaces are already somewhat aligned by the frozen models, which may not hold for distant language pairs or highly specialized domains. The paper does not address how the system handles very long context sequences (computational complexity), how it handles code-switching (mixed-language utterances), or how it performs on accents underrepresented in the pretraining data. Ablation studies on the projection module design (bilinear vs. cross-attention) and context window size limits are missing from the abstract.
Research Context
This work builds on a wave of recent advances: (1) frozen-pretrained-model architectures (e.g., Flamingo, BLIP-2) that achieve strong performance with minimal fine-tuning; (2) context-aware NLP methods that use in-context learning and prompt engineering rather than full retraining (pioneered by large language models); (3) multilingual ASR systems like Whisper and XLS-R that proved one model can span many languages. The paper extends context-aware ASR beyond isolated utterances—prior work like RNN-T with contextual biasing handled simple word lists, not rich dialogue history. It likely benchmarks on multilingual speech datasets (Common Voice, BABEL, or similar) and evaluates on domain adaptation tasks where context should matter most (e.g., customer service transcription with dialogue, medical notes with biasing words). This opens a research direction: can similar modular projection approaches work for other speech tasks (speaker diarization, speech translation) and other modalities (vision-language with frozen CLIP encoders)?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::