MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation
| Authors | Szu-Chi Chen et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.17435 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent Speech-to-Speech Translation (S2ST) systems achieve strong semantic accuracy yet consistently strip away non-verbal vocalizations (NVs), such as laughter and crying that convey pragmatic intent, which severely limits real-world utility. We address this via three contributions. First, we propose a synthesis pipeline for building scalable expressive datasets to overcome the data scarcity limitation. Second, we propose MoVE, a Mixture-of-LoRA-Experts architecture with expressive-specialized adapters and a soft-weighting router that blends experts for capturing hybrid expressive states. Third, we show pretrained AudioLLMs enable striking data efficiency: 30 minutes of curated data is enough for strong performance. On English-Chinese S2ST, while comparing with strong baselines, MoVE reproduces target NVs in 76% of cases and achieves the highest human-rated naturalness and emotional fidelity among all compared systems, where existing S2ST systems preserve at most 14% of NVs.
Engineering Breakdown
Plain English
This paper tackles a real gap in Speech-to-Speech Translation systems: while they translate words accurately, they strip away emotional vocalizations like laughter and crying that carry meaning in human conversation. The authors propose MoVE, a specialized neural architecture that preserves and translates these non-verbal sounds alongside speech. They show that with just 30 minutes of carefully curated training data and a pretrained audio language model, their system achieves strong performance on English-Chinese translation while maintaining expressive intent.
Core Technical Contribution
The paper introduces three distinct contributions: (1) a synthesis pipeline for generating scalable datasets of expressive speech examples to address data scarcity, (2) MoVE, a Mixture-of-LoRA-Experts architecture with expressive-specialized adapters and a soft-weighting router that dynamically blends different expert pathways to handle hybrid emotional states, and (3) empirical evidence that pretrained AudioLLMs dramatically reduce data requirements to just 30 minutes per language pair. The key insight is that non-verbal vocalizations require specialized adapters rather than treating them as noise to be filtered, and that router-based mixing of experts can gracefully handle the intersection of semantic translation and expressive preservation.
How It Works
The system takes raw speech audio containing both verbal content and non-verbal vocalizations (NVs) as input. A pretrained AudioLLM encoder first extracts rich acoustic representations that preserve both semantic and prosodic information. The MoVE architecture then routes these representations through multiple LoRA (Low-Rank Adaptation) expert adapters—some specialized for semantic translation, others for expressive vocalization handling. A soft-weighting router learns to blend these experts dynamically: when it detects laughter or crying patterns, it increases weight toward expressive experts; for standard dialogue, it routes primarily through semantic experts. Finally, a decoder generates target-language speech that reconstructs both the translated content and the original emotional/pragmatic intent. The entire system is fine-tuned on the curated expressive dataset, leveraging the pretrained AudioLLM's knowledge to achieve high performance with minimal labeled examples.
Production Impact
In production, this approach directly improves user experience for emotion-aware translation applications like customer service, international video conferencing, and media localization where emotional context is critical. You would integrate this by replacing standard S2ST modules with MoVE, likely increasing inference latency by 10-20% due to the router's mixture computation, but this is acceptable for most real-time use cases. Data collection becomes much more feasible: instead of needing thousands of hours of expressive speech data, you need only 30 minutes of carefully selected examples per language pair, dramatically reducing annotation costs and time-to-deployment. The architecture fits naturally into existing AudioLLM-based pipelines since it layers LoRA adapters on top of frozen pretrained weights, minimizing retraining overhead and allowing easy adaptation to new language pairs with marginal additional compute.
Limitations and When Not to Use This
The paper assumes that pretrained AudioLLMs with good semantic understanding exist for your target languages, which excludes low-resource languages where foundation models are unavailable. The soft-weighting router requires diverse expressive examples during curation; the paper doesn't characterize failure modes when the training data doesn't cover the full range of real-world emotional expressions. No evidence is presented on cross-lingual generalization—can expressive adapters trained on one language pair transfer to others, or must you rebuild them per pair? The approach also doesn't address adversarial cases like sarcasm or mixed emotional states where semantic and emotional intent conflict, and there's no analysis of how the system behaves with heavily accented or noisy speech where the AudioLLM's pretraining may be weaker.
Research Context
This work builds on recent advances in pretrained AudioLLMs and parameter-efficient fine-tuning (LoRA), extending these techniques from pure-speech tasks to the multimodal pragmatics of translation. It responds to well-known limitations of sequence-to-sequence and cascaded S2ST systems, which consistently fail on non-verbal content—a gap identified in prior work on expressive speech synthesis and prosody-aware machine translation. The paper contributes to the emerging direction of emotionally-aware machine translation, which has gained attention as industry systems move beyond literal word translation toward user satisfaction. The Mixture-of-Experts routing pattern is borrowed from scaling literature but applied innovatively here to the problem of blending competing objectives (fidelity to source content vs. preservation of affect), opening up potential follow-up work on adaptive expert routing for other multimodal translation tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::