Make It Hard to Hear, Easy to Learn: Long-Form Bengali ASR and Speaker Diarization via Extreme Augmentation and Perfect Alignment
| Authors | Sanjid Hasan et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2602.23070 |
| Download | |
| Categories | cs.SD, cs.AI, cs.CL |
Abstract
Although Automatic Speech Recognition (ASR) in Bengali has seen significant progress, processing long-duration audio and performing robust speaker diarization remain critical research gaps. To address the severe scarcity of joint ASR and diarization resources for this language, we introduce Lipi-Ghor-882, a comprehensive 882-hour multi-speaker Bengali dataset. In this paper, detailing our submission to the DL Sprint 4.0 competition, we systematically evaluate various architectures and approaches for long-form Bengali speech. For ASR, we demonstrate that raw data scaling is ineffective; instead, targeted fine-tuning utilizing perfectly aligned annotations paired with synthetic acoustic degradation (noise and reverberation) emerges as the singular most effective approach. Conversely, for speaker diarization, we observed that global open-source state-of-the-art models (such as Diarizen) performed surprisingly poorly on this complex dataset. Extensive model retraining yielded negligible improvements; instead, strategic, heuristic post-processing of baseline model outputs proved to be the primary driver for increasing accuracy. Ultimately, this work outlines a highly optimized dual pipeline achieving a 0.019 Real-Time Factor (RTF), establishing a practical, empirically backed benchmark for low-resource, long-form speech processing.
Engineering Breakdown
Plain English
This paper tackles the problem of processing long-form Bengali speech for both automatic speech recognition (ASR) and speaker diarization—two related but distinct tasks that have received little joint attention in low-resource languages. The authors introduce Lipi-Ghor-882, an 882-hour multi-speaker Bengali dataset, and demonstrate that naive data scaling doesn't help ASR performance; instead, using perfectly aligned annotations combined with synthetic acoustic degradation (adding noise and reverberation to training data) is far more effective. The work shows that speaker diarization benefits from a different strategy than ASR, suggesting these tasks require fundamentally different optimization approaches despite their overlapping signal processing foundations.
Core Technical Contribution
The core contribution is empirically demonstrating that for low-resource Bengali ASR, data quantity alone is insufficient—what matters is annotation quality and strategic acoustic augmentation. The authors challenge the common assumption that 'more data is always better' by showing that perfectly aligned training labels paired with synthetic noise/reverberation substantially outperforms simply increasing dataset size. They also reveal a task-specific divergence: speaker diarization requires a different training recipe than ASR, even when both operate on the same audio input. Finally, the release of Lipi-Ghor-882 provides the community with a rare joint ASR+diarization resource for Bengali, addressing a documented scarcity of such paired datasets.
How It Works
The pipeline ingests long-form Bengali audio and applies two parallel processing branches. For ASR, the system first applies synthetic acoustic degradation to raw audio—injecting noise and reverberation at training time to make the model robust to real-world conditions (the 'hard to hear' phase). The degraded audio is then fed to an ASR encoder-decoder model trained with perfectly aligned annotations, where 'perfect alignment' means manual verification that every word boundary and phonetic unit is precisely labeled (the 'easy to learn' phase). For speaker diarization, a separate branch processes the same audio through a speaker embedding extractor and clustering module, likely using different augmentation or loss functions than ASR. The authors evaluate multiple architectural choices for the encoder backbone (likely Transformer variants like Whisper or Conformer) and find that the combination of high-quality labels + synthetic augmentation consistently yields the best character error rate (CER) for ASR, while diarization appears to respond better to alternative optimization strategies—suggesting the two tasks have different gradient landscapes despite sharing input representations.
Production Impact
For engineers deploying ASR systems in Bengali or other low-resource languages, this work fundamentally changes the ROI calculus: instead of spending effort collecting raw unlabeled audio, allocate resources to perfect annotation quality and synthetic degradation pipelines. In a real production system, this means (1) investing in careful data labeling and QA workflows with native speakers, (2) implementing a multi-stage training pipeline that first adds synthetic noise/reverb then fine-tunes on clean data, and (3) recognizing that speaker diarization may need its own separate model or training procedure rather than sharing ASR weights. The latency impact is minimal—synthetic augmentation happens once during training, not at inference. However, the compute cost increases during training due to augmentation preprocessing and longer fine-tuning schedules. For teams with limited Bengali speech data, this approach makes adoption feasible: the Lipi-Ghor-882 dataset provides a concrete starting point, eliminating months of data collection work.
Limitations and When Not to Use This
The paper does not clearly address how performance scales beyond 882 hours or whether the augmentation strategy generalizes to other low-resource languages with different phonetic properties—Bengali's use of complex consonant clusters and retroflex sounds may require language-specific augmentation tuning. The abstract cuts off mid-sentence for speaker diarization findings, leaving unclear what strategy actually works best for that task, which limits reproducibility. The work assumes access to native speakers for perfect annotation, a constraint many low-resource language projects cannot meet; it doesn't explore graceful degradation with partial or crowd-sourced labels. There is no discussion of computational costs for the fine-tuning pipeline or inference latency requirements, making it difficult to assess feasibility for real-time or edge deployment scenarios common in mobile ASR applications.
Research Context
This work builds on the established finding that data quality beats data quantity in low-resource speech recognition, extending prior work (like Whisper's multilingual training) by showing this principle applies even more sharply in the long-form, speaker-heavy domain. The paper contributes to a growing body of research on non-English and morphologically-rich language ASR—Bengali joins a limited set of languages with dedicated long-form datasets comparable to English benchmarks like LibriSpeech. The synthetic augmentation approach leverages techniques from speech enhancement and domain adaptation literature, applying them in a structured way to maximize robustness. By releasing Lipi-Ghor-882, this work enables future research on multilingual diarization and code-switched speech (relevant for Bengali communities in diaspora), potentially influencing how other South Asian languages tackle these paired tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::