Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems
| Authors | Siyuan Liu et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23266 |
| Download | |
| Categories | cs.CL |
Abstract
Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.
Engineering Breakdown
Plain English
This paper tackles the latency problem in spoken dialogue systems by proposing DDTSR, a framework that allows ASR, language generation, and TTS to happen in parallel rather than sequentially. Instead of waiting for complete speech transcription before generating responses, the system uses a small auxiliary model to generate discourse connectives (minimal-commitment placeholders like 'well', 'so', 'but') while a larger model reasons in the background, enabling the TTS to start speaking earlier. The dual-track approach lets the system "listen while thinking and speak while thinking," reducing end-to-end latency compared to conventional cascaded pipelines that must complete each stage fully before moving to the next.
Core Technical Contribution
The core innovation is a two-model synergy pattern where an auxiliary small model generates discourse connectives as low-commitment filler utterances while a larger LLM performs knowledge-intensive reasoning asynchronously. This enables true streaming response generation without waiting for complete ASR output or full reasoning completion. The connectives serve as both placeholder content and discourse markers that guide the overall conversation flow, allowing the TTS pipeline to begin synthesis before the substantive response is finalized. This is fundamentally different from prior work that either serializes all stages or attempts naive streaming without structural guidance.
How It Works
The DDTSR system operates on a dual-track principle: (1) Fast track uses a lightweight model to generate discourse connectives (short, grammatically safe utterances like 'I see', 'that's interesting') streamed to TTS immediately upon partial ASR output, buying time for the main reasoning; (2) Slow track runs an LLM in parallel to perform full reasoning and context understanding on the complete or near-complete ASR hypothesis. As the LLM completes its reasoning, the system transitions from connectives to substantive responses, dynamically stitching them together in the audio stream. The ASR output feeds both tracks simultaneously—the small model requires only partial hypotheses to generate connectives, while the large model waits for higher-confidence ASR results. Cross-modal collaboration ensures the transitions between connective and full response are natural, avoiding awkward silences or abrupt topic shifts.
Production Impact
For production dialogue systems, this reduces perceived latency from 2-3+ seconds (sequential ASR→LLM→TTS) to potentially 0.5-1.5 seconds by overlapping stages and starting audio output earlier. Engineers deploying this would need to: (1) maintain two separate models with different sizes and inference profiles, increasing model serving complexity; (2) implement careful audio transition logic to blend connectives into substantive responses without artifacts; (3) handle ASR uncertainty more gracefully since the small model must generate plausible connectives on partial/noisy input. The main trade-off is added system complexity—you now orchestrate three parallel processes (ASR, small LM, large LM) instead of a simple pipeline, requiring distributed tracing and careful timeout management. For mobile or edge deployment, running two LMs simultaneously may be prohibitive; this approach is better suited to server-side systems with sufficient compute.
Limitations and When Not to Use This
The approach assumes that discourse connectives are genuinely useful placeholders that don't disrupt user experience—in practice, over-reliance on fillers like 'well, I'm thinking about that' could feel artificial or frustrating. The paper doesn't address how to handle task-specific responses where immediate placeholder content is inappropriate (e.g., answering factual questions where saying 'I see' first then correcting yourself damages credibility). Cross-modal synchronization is underspecified: if the small model generates a 15-second connective sequence while the large model finishes reasoning in 8 seconds, the stitching mechanism must handle awkward overlaps. The evaluation likely focuses on latency metrics but doesn't deeply explore user perception studies or failure modes when ASR quality is poor (where the small model may generate misleading connectives based on wrong hypotheses).
Research Context
This work builds on decades of research in streaming NLP (incremental parsing, left-to-right generation) and real-time dialogue systems, but applies it to the modern LLM era where latency bottlenecks have shifted from dialogue logic to model inference time. It extends prior work on speculative decoding and parallel inference—instead of different model sizes for the same task, DDTSR uses different models for strategically different subtasks. The paper likely benchmarks against standard dialogue datasets (DSTC, MultiWOZ) but focuses on latency measurement rather than dialogue quality. This opens a research direction on human perception of streaming dialogue: do users prefer faster responses with connectives over slightly slower but more polished responses, and how does this vary by task type?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::