VoxMind: An End-to-End Agentic Spoken Dialogue System
| Authors | Tianle Liang et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2604.15710 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.
Engineering Breakdown
Plain English
VoxMind is an end-to-end spoken dialogue system that extends conversational AI models with tool-use and agentic capabilities—the ability to reason about tasks, call external APIs, and solve complex real-world problems beyond pure conversation. The authors trained on a curated 470-hour AgentChat dataset and introduced a 'Think-before-Speak' mechanism that makes the model plan its response before generating speech, similar to chain-of-thought reasoning in text models. This addresses a real gap: existing spoken dialogue systems are good at talking but struggle when users ask them to do things—book flights, answer domain-specific questions, or coordinate multi-step tasks. The framework unifies speech understanding, agentic reasoning, and speech generation in one end-to-end system.
Core Technical Contribution
The core novelty is the 'Think-before-Speak' mechanism: an intermediate reasoning layer inserted between the speech encoder and the speech decoder that performs explicit planning before committing to an output. Unlike prior work that either bolts tools onto frozen language models or treats tool use as a separate downstream module, VoxMind integrates agent reasoning directly into the spoken dialogue pipeline, allowing the model to reason about which tools to use while maintaining speech-native input/output. The 470-hour AgentChat dataset, curated for tool-augmented dialogue, is a second significant contribution—it includes conversational turns paired with tool calls, execution results, and corresponding speech outputs. This is different from prior approaches because it addresses the full pipeline from acoustic input through reasoning to acoustic output, rather than treating speech as a wrapper around text-based agents.
How It Works
The system accepts raw audio input, which is encoded by a speech encoder into a continuous representation (similar to a spectrogram embedding or acoustic tokens). This encoding is fed into an agentic reasoning layer—the 'Think' phase—where the model generates an internal plan, decides which tools to call, and reasons about the task semantically without yet committing to speech. Once the reasoning is complete, the model transitions to the 'Speak' phase, where the plan is decoded into natural language and then synthesized into speech via a speech decoder. When a tool call is needed (e.g., querying a database), the system interrupts the speech generation, executes the tool in the background, and conditions the subsequent speech generation on the tool's output. The entire pipeline is trained end-to-end on the AgentChat dataset using a combination of supervised learning (to match ground-truth tool calls and outputs) and likely reinforcement learning objectives (to maximize task success and speech naturalness).
Production Impact
For teams building conversational AI systems, VoxMind eliminates the need to stitch together separate ASR → NLU → planning → dialogue → TTS pipelines, reducing latency and engineering complexity. If you adopt this approach, you can handle complex, multi-step user requests directly through voice without dropping to a text intermediary or losing context across modalities. The trade-off is significant: you need 470 hours of high-quality, tool-augmented training data (roughly 1,690 hours of speech if typical density assumptions hold), which is expensive to collect and curate. Inference latency will increase due to the intermediate reasoning layer—estimates suggest 200-500ms additional latency compared to vanilla dialogue systems—because the model must reason before speaking rather than streaming output immediately. Integration requires careful system design: you must define your tool interface clearly, handle tool execution timeouts and failures gracefully, and log all tool calls for safety and debugging. The compute footprint during training and inference is substantially higher than text-only agents, though the paper doesn't specify exact FLOP counts or GPU memory requirements.
Limitations and When Not to Use This
The paper's scope is limited to English-only dialogue (no evidence of multilingual generalization), and the AgentChat dataset may not represent the full distribution of real-world tool calls users actually request—particularly edge cases and adversarial inputs. The 'Think-before-Speak' mechanism assumes the model can accurately reason about task structure before generating output, but there's no analysis of failure modes when the reasoning layer makes incorrect tool-selection decisions; there's also no explicit safety analysis for cases where tool outputs contradict the user's intent or contain sensitive data. The approach requires end-to-end training, which means you cannot easily plug this system into existing, frozen language models or reasoning engines—retraining from scratch on your own domain is necessary, raising the barrier to adoption for smaller teams. Finally, the paper doesn't report comparative benchmarks against baselines (e.g., traditional modular pipelines or other agentic spoken-dialogue systems), so it's unclear whether the integrated approach actually outperforms simpler alternatives in real-world task success rates or user satisfaction.
Research Context
This work builds on the recent success of end-to-end neural dialogue models (influenced by transformer-based seq2seq speech systems like Whisper and neural TTS systems like Tacotron 2) and extends them with tool-use capabilities that have become standard in LLM-based agents (e.g., ReAct, chain-of-thought prompting). It directly addresses a gap identified in the broader spoken-dialogue community: while text-based LLM agents have been extensively studied for tool use, the speech modality has been largely overlooked. The AgentChat dataset joins a growing body of agent-centric training corpora (similar to toolbench or ToolLLM datasets) but specialized for speech. This opens a research direction around 'multimodal agentic reasoning'—the intersection of spoken interaction, tool use, and real-time task execution—which is likely to become a major focus as voice interfaces become more pervasive in enterprise and consumer applications.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::