Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text
| Authors | Hainan Xu et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.24245 |
| Download | |
| Categories | cs.LG |
Abstract
We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.
Engineering Breakdown
Plain English
This paper introduces Chunk-wise Attention Transducers (CHAT), which extends RNN-T models for streaming speech-to-text by processing audio in fixed-size chunks with cross-attention mechanisms within each chunk. The approach maintains the streaming capability of RNN-T while dramatically improving efficiency: 46.2% reduction in peak training memory, 1.36X faster training, and 1.69X faster inference. Beyond efficiency, CHAT achieves meaningful accuracy improvements across multiple languages and tasks, including up to 6.3% relative WER reduction for speech recognition and 18.0% BLEU improvement for speech translation. This combines the real-time properties necessary for production systems with the accuracy improvements that matter for user-facing applications.
Core Technical Contribution
The core novelty is the chunk-wise processing strategy that breaks the RNN-T's standard token-by-token streaming constraint without sacrificing true online capability. Instead of processing individual audio frames sequentially, CHAT groups them into fixed-size chunks and applies cross-attention within each chunk to model local alignment patterns more flexibly than RNN-T's label-driven alignment mechanism. This reduces the temporal dimension that the RNN-T's transducer joint network must handle, which is the computational bottleneck in standard RNN-T. The architectural contribution is relatively simple but effective: it's a disciplined way to introduce more context into alignment modeling while preserving the streaming property that keeps end-to-end latency low.
How It Works
CHAT processes incoming audio streams in fixed-size chunks (the paper doesn't specify chunk size, but this is a critical hyperparameter). Each chunk undergoes standard acoustic encoding (likely with a standard CNN-RNN or Transformer encoder), producing a sequence of acoustic representations. Within each chunk, a cross-attention mechanism allows these representations to directly attend to one another, creating local context for alignment decisions before the joint network processes them. The joint network (which combines acoustic and label embeddings) then processes each attended chunk output sequentially, maintaining the streaming RNN-T structure. The key insight is that by allowing intra-chunk attention, you reduce the burden on the joint network's hidden state to capture all alignment information — it only needs to handle alignment between chunks, not within them. This separation of concerns (local alignment via attention, cross-chunk alignment via RNN-T's sequential state) produces both the efficiency and accuracy gains.
Production Impact
For teams running speech-to-text or speech-to-translation systems in production, CHAT offers a direct upgrade path that improves both cost and quality simultaneously. The 46.2% reduction in peak training memory means you can train larger models on existing hardware or train on cheaper, smaller GPU instances, reducing infrastructure costs significantly. The 1.69X inference speedup directly reduces latency per user request and increases throughput per server, which translates to lower cloud compute costs and better user experience for real-time transcription features. However, integrating CHAT requires retraining existing RNN-T checkpoints from scratch (no straightforward fine-tuning path from existing models is mentioned), which creates deployment friction during the transition. The accuracy improvements (up to 6.3% WER reduction) are substantial enough to justify retraining for many teams, but you'll need to validate that the chunk-based processing doesn't introduce latency artifacts in your specific audio domain or language combination.
Limitations and When Not to Use This
The paper does not clarify the chunk size selection process or provide ablations showing sensitivity to this hyperparameter, which is critical for practitioners. There's no discussion of how CHAT handles variable-length sequences or edge cases like very short utterances that might not fill a full chunk, which matters for real-world robustness. The streaming property is preserved theoretically, but there's no analysis of actual end-to-end latency (only training/inference speed), so it's unclear whether chunk-wise processing introduces noticeable latency increases in ultra-low-latency applications (e.g., live simultaneous interpretation). The paper also doesn't address how CHAT performs on out-of-domain data or noisy audio, which are common failure modes in production; the improvements are demonstrated on benchmarks that may not reflect your specific data distribution. Finally, there's no discussion of how to extend CHAT to other transducer variants or whether the approach is compatible with other recent RNN-T improvements like non-auto-regressive decoding.
Research Context
CHAT builds directly on the RNN-T framework (Graves 2012, Graves et al. 2020), which is the dominant streaming speech recognition architecture in production systems at Google, Amazon, and others. The paper is positioned as addressing the well-known computational bottleneck of RNN-T's joint network, which has been a limiting factor for deploying large models in real-time applications. The use of chunk-wise processing and intra-chunk attention connects to broader trends in efficient sequence modeling (e.g., local attention in Transformers, Longformer-style chunking), applying these ideas to the RNN-T setting. The multi-task evaluation across speech recognition and speech translation on standard benchmarks (likely LibriSpeech, must-c, or similar) demonstrates broad applicability, positioning CHAT as a general architectural improvement rather than a task-specific hack. This work opens the door for further exploration of hybrid attention-RNN approaches in streaming settings, potentially inspiring similar chunk-wise decompositions in other streaming architectures.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::