MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
| Authors | He Zhang et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2511.10262 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench
Engineering Breakdown
Plain English
This paper introduces MTR-DuplexBench, a new evaluation benchmark for Full-Duplex Speech Language Models (FD-SLMs) that enable real-time overlapping conversations, moving beyond single-round interaction testing. The authors identified a critical gap: existing benchmarks don't properly evaluate multi-round conversational scenarios where turn boundaries blur and context consistency breaks down during inference. MTR-DuplexBench addresses this by providing comprehensive multi-round evaluation capabilities that test conversational features alongside other critical aspects of FD-SLM performance. The work enables researchers and practitioners to rigorously assess how well these models handle the complexities of natural, continuous dialogue.
Core Technical Contribution
The core novelty is MTR-DuplexBench itself—a purpose-built benchmark that specifically handles the technical challenges of evaluating full-duplex speech interactions across multiple turns. Unlike prior single-round benchmarks, this framework explicitly models blurred turn boundaries (where speakers overlap or interrupt) and tracks context consistency as the model processes sequential exchanges. The benchmark extends evaluation beyond conversational metrics to assess other critical dimensions of FD-SLM behavior that holistic production systems require. This represents the first systematic approach to multi-round full-duplex speech evaluation, filling a concrete gap in how these emerging systems can be rigorously tested.
How It Works
MTR-DuplexBench operates by constructing multi-turn dialogue sequences that intentionally include realistic conversational phenomena like overlapping speech, interruptions, and context drift across turns. The benchmark takes as input raw speech interactions (potentially with multiple speakers active simultaneously) and passes them through the FD-SLM being evaluated. The model must maintain coherent internal context across turns while handling the blurred turn boundaries that naturally occur in full-duplex conversation—distinguishing itself from turn-based systems where one speaker finishes completely before the other begins. The evaluation framework then measures model outputs across multiple dimensions: conversational quality, context preservation accuracy, turn-boundary handling correctness, and other critical performance metrics that go beyond traditional accuracy measures. Crucially, the benchmark tracks how well the model's inference state remains consistent as it processes overlapping or rapidly-switching dialogue, exposing failure modes that single-round evaluation would miss. The segmentation of continuous speech into meaningful evaluation units solves the technical problem of knowing where dialogue boundaries exist when speakers overlap.
Production Impact
For engineers building real-world conversational AI systems, MTR-DuplexBench enables rigorous quality assurance before deployment, replacing informal testing with systematic multi-turn evaluation. This directly addresses a production pain point: current benchmarks certify models on artificial single-exchange scenarios, but real users engage in continuous multi-turn conversations where context degradation and turn-handling failures accumulate and become noticeable. Adopting this benchmark in your evaluation pipeline means catching context-consistency bugs, turn-boundary errors, and degradation patterns that only emerge across 5+ turn interactions—problems that would otherwise surface as user-reported quality issues post-launch. The tradeoff is modest: evaluation time increases because you're testing longer dialogue sequences and multiple quality dimensions rather than single turns, but this cost is negligible compared to the cost of deploying models that fail on real conversational patterns. Integration is straightforward for teams already using speech benchmarks—it's additive, expanding your test suite rather than replacing existing infrastructure.
Limitations and When Not to Use This
The paper doesn't address how FD-SLMs should be trained or architected to handle these multi-round scenarios—it's purely an evaluation framework, not a training methodology or architectural innovation. The benchmark likely requires carefully curated multi-turn dialogue data, raising questions about dataset availability, annotation cost, and whether the benchmark's conversational patterns generalize across languages, accents, and domains beyond the test set. MTR-DuplexBench also doesn't solve the fundamental inference latency challenge of full-duplex systems: having a benchmark doesn't reduce the computational cost of maintaining context and generating responses while processing overlapping input speech in real-time. The paper's focus on conversational correctness doesn't directly address safety concerns specific to full-duplex systems—for instance, what happens when overlapping speakers discuss sensitive topics or the model misinterprets which speaker said what during interruptions, potentially leading to safety failures the benchmark may not expose.
Research Context
This work builds directly on the emerging field of full-duplex conversational AI, extending recent advances in speech language models toward more natural, simultaneous interaction patterns. It addresses a documented limitation in existing speech benchmarks (which typically measure single-turn ASR accuracy or isolated dialogue exchanges) by introducing the first multi-round, overlap-aware evaluation framework. The research opens the door to systematic comparison of FD-SLM architectures and training approaches, enabling the field to move beyond anecdotal performance claims. MTR-DuplexBench sits alongside other recent benchmarks in conversational AI (like those for turn-taking, interruption handling, and context tracking) but uniquely focuses on the full-duplex, overlapping-speech scenario that mirrors human conversation more closely than traditional dialogue systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::