LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation
| Authors | Venkata Pushpak Teja Menta |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2605.00777 |
| Download | |
| Categories | cs.SD, cs.CL |
Abstract
A speaker encoder used in multilingual voice cloning should treat the same speaker identically regardless of which script the audio was uttered in. Off-the-shelf encoders do not, and the failure is accent-conditional. On a 1043-pair Western-accented voice corpus across English, Hindi, Telugu, and Tamil, WavLM-base-plus-sv loses 0.082 absolute cosine similarity when the same voice changes script and ECAPA-TDNN loses 0.105. On a 1369-pair Indian-accented voice corpus, the gap shrinks to 0.006 (WavLM-SV) and 0.044 (ECAPA-TDNN). The leak is largest where it matters most for cross-script TTS: when a system projects a non-Indic-trained voice into Indic scripts. We present LASE (Language-Adversarial Speaker Encoder), a small projection head over frozen WavLM-base-plus trained with two losses: a supervised contrastive loss over voice identity, and a gradient-reversal cross-entropy against a 4-language classifier that pushes the embedding to be language-uninformative while remaining speaker-informative. Trained on 1118 quality-gated cross-script pairs synthesised from 8 commercial multilingual voices, LASE's residual gap is consistent with zero on both corpora (Delta = 0.013 Western, Delta = 0.026 Indian; both bootstrap 95% CIs include zero) and amplifies the cross-script-vs-floor margin 2.4-2.7x over both baselines. An ECAPA+GRL ablation shows the GRL objective improves either backbone but the WavLM choice contributes too. In synthetic multi-speaker diarisation, LASE matches ECAPA-TDNN on cross-script speaker recall (0.788 vs 0.789) with ~100x less training data. We release the r1 checkpoint, both corpora, and the bootstrap recipe.
Engineering Breakdown
Plain English
This paper identifies a critical failure in off-the-shelf speaker encoders when used for multilingual voice cloning: they don't treat the same speaker identically across different languages/scripts. On Western-accented voices, WavLM-base-plus loses 0.082 cosine similarity and ECAPA-TDNN loses 0.105 when the same speaker switches from English to Hindi/Telugu/Tamil. The problem shrinks on Indian-accented voices but remains significant when projecting non-Indic-trained voices into Indic scripts. The authors propose LASE (Language-Adversarial Speaker Encoder), a learnable projection head trained with adversarial losses on frozen WavLM-base-plus to decouple speaker identity from language/script identity.
Core Technical Contribution
The core contribution is identifying and quantifying accent-conditional speaker encoder drift across scripts—a problem that prior encoders didn't acknowledge or measure systematically. The technical novelty is LASE's two-loss adversarial training approach: a speaker verification loss that keeps speaker embeddings close across scripts, paired with a language adversarial loss that prevents a discriminator from predicting the script/language from those embeddings. Unlike full model fine-tuning, LASE keeps WavLM frozen and only trains a small projection head, making it parameter-efficient and practical for production deployment. The insight that the failure is most severe where it matters most (Western accents speaking Indic scripts) directly shapes the remedy.
How It Works
The system takes audio input in any script and passes it through a frozen WavLM-base-plus encoder to get a speaker representation. This representation then flows through a small learnable projection head (LASE), which produces a speaker embedding. During training, two objectives run in parallel: (1) a speaker verification loss (triplet or contrastive) that pulls embeddings from the same speaker closer together, regardless of which script they speak, and (2) a language adversarial loss where a script discriminator tries to predict which language/script produced the embedding, while the projection head is trained to fool it (gradient reversal). The frozen backbone ensures the model preserves WavLM's multilingual and acoustic understanding, while the adversarial head learns to suppress language-specific information in the speaker embedding space. At inference, only the frozen encoder and small projection head are needed—no discriminator.
Production Impact
For teams building multilingual voice cloning systems, LASE directly solves the speaker leakage problem that degrades voice consistency across language switches. In a production TTS pipeline, you'd replace your speaker encoder with this frozen-WavLM-plus-LASE setup—a simple drop-in with minimal latency overhead (just a small projection matrix multiply). The training cost is low: you only fine-tune the projection head (likely <100K parameters) with adversarial losses on multilingual speaker pairs, not the full 300M+ parameter WavLM model. Concretely, if you're cloning a Western English speaker into Hindi, the voice will stay recognizable instead of drifting; similarly for Indian English speakers across Indic scripts. The trade-off is that you need multilingual paired speaker data to train the adversarial losses—synthetic or crowdsourced multilingual utterances from the same speakers.
Limitations and When Not to Use This
The paper's evaluation is limited to four languages (English, Hindi, Telugu, Tamil) and two accent groups (Western and Indian), so generalization to other language families, tonal languages, or low-resource scripts remains unproven. LASE assumes the underlying WavLM encoder is already reasonably multilingual—it won't rescue an encoder trained only on English. The paper does not compare against other potential solutions like data augmentation, joint training on multilingual data, or full model fine-tuning; it's unclear how much of the improvement is due to the adversarial approach versus simply training a projection head on diverse multilingual data. The human evaluation of naturalness and speaker consistency in cloned voices is missing from the abstract, so it's unclear if the cosine similarity improvement translates to perceptually better clones in production.
Research Context
This work builds on the established speaker encoder literature (WavLM, ECAPA-TDNN, speaker verification) and applies adversarial debiasing techniques borrowed from fairness/domain adaptation research. It directly addresses a gap in multilingual voice conversion and TTS, where script-invariant speaker embeddings have been implicitly assumed to exist but never validated empirically. The paper contributes a benchmark—the 1043-pair Western and 1369-pair Indian multilingual corpus with clear accent and script labels—that enables future work on language-invariant speaker representations. This opens a research direction in accent-aware and script-aware speaker modeling, relevant for low-resource language TTS and cross-lingual voice transfer.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::