The TTS-STT Flywheel: Synthetic Entity-Dense Audio Closes the Indic ASR Gap Where Commercial and Open-Source Systems Fail
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-04 with 2 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Venkata Pushpak Teja Menta |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2605.03073 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Niche-domain Indic ASR -- digit strings, currency amounts, addresses, brand names, English/Indic codemix -- is under-served by both open-source SOTA and commercial systems. On a synthesised entity-dense Telugu test set (held-out by synthesis system), vasista22/whisper-telugu-large-v2 (open SOTA) achieves Entity-Hit-Rate (EHR) 0.027 and Deepgram Nova-3 (commercial) 0.16. We close this gap with a self-contained TTS<->STT flywheel: an open-source Indic TTS pipeline synthesises ~22,000 entity-dense Indic-English code-mix utterances at <$50 marginal cost, and a LoRA fine-tune on top of vasista22 achieves EHR 0.473 on the held-out test (17x over open SOTA, 3x over commercial), with read-prose regression bounded to +6.6 pp WER on FLEURS-Te. Cross-language: beta-Hi 0.337 (7x vs vasista22) and beta-Ta 0.543 (22x vs vasista22, 22x vs Deepgram); on Hindi where Deepgram has substantial entity coverage, the flywheel underperforms commercial. All three beta models fall below pre-registered EHR targets (0.75 for Te, 0.65 for Hi/Ta); we report honestly. A native-human-recorded sanity check (n=20 Telugu) confirms transfer to real speech (beta-Te EHR 0.516 on native vs 0.473 on synth). An EDSA-isolation ablation (LoRA on FLEURS-Te alone) yields EHR 0.020 on the same held-out, attributing ~100% of the gain to the EDSA corpus. We additionally report a language-conditional finding: vanilla Whisper-large-v3 has Telugu-specific Script Collapse (SFR 0.46-0.71) that a per-language LoRA corrects (SFR 0.81-0.97), but the recipe is contraindicated on Hindi and Tamil where vanilla SFR >= 0.98. Code, holdouts, predictions, EDSA corpus, and entity dictionaries are released open-source.
Engineering Breakdown
Plain English
This paper solves a real problem: existing speech-to-text systems (both open-source and commercial) are terrible at recognizing entity-dense Indic language utterances like phone numbers, addresses, and brand names mixed with English. The authors built a closed-loop system where they use an open-source TTS pipeline to generate ~22,000 synthetic training examples for $50, then fine-tune an existing Telugu ASR model with LoRA, achieving 47.3% entity recognition accuracy — 17x better than the open-source baseline and 3x better than Deepgram's commercial system.
Key Engineering Insight
The flywheel pattern — using TTS to generate synthetic, entity-dense training data specifically designed to target the weaknesses of existing STT systems — is dramatically more cost-effective than collecting real audio. For under $50 in compute, they got 17x improvement on a task where commercial systems were failing, proving that targeted synthetic data beats scale when you understand the failure mode.
Why It Matters for Engineers
Production systems serving Indic language users hit a wall with entities: phone numbers, amounts, addresses all fail silently. This paper shows a practical, cheap playbook for fixing task-specific ASR gaps without large labeled datasets or expensive commercial licensing. It's directly applicable to any niche domain where you need reliable entity extraction in speech.
Research Context
Prior work struggled with Indic ASR generally; this paper shifts focus to the specific, practical problem of entity recognition in code-mixed speech. It advances the TTS-as-training-data synthesis concept by closing a concrete gap that commercial systems leave open. This enables small teams to build production-grade ASR for underserved languages by borrowing existing TTS/STT infrastructure and LoRA fine-tuning — no novel architectures required.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::