A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversations
| Authors | Soumya Dutta et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2602.23300 |
| Download | |
| Categories | cs.CL |
Abstract
Emotion Recognition in Conversations (ERC) presents unique challenges, requiring models to capture the temporal flow of multi-turn dialogues and to effectively integrate cues from multiple modalities. We propose Mixture of Speech-Text Experts for Recognition of Emotions (MiSTER-E), a modular Mixture-of-Experts (MoE) framework designed to decouple two core challenges in ERC: modality-specific context modeling and multimodal information fusion. MiSTER-E leverages large language models (LLMs) fine-tuned for both speech and text to provide rich utterance-level embeddings, which are then enhanced through a convolutional-recurrent context modeling layer. The system integrates predictions from three experts-speech-only, text-only, and cross-modal-using a learned gating mechanism that dynamically weighs their outputs. To further encourage consistency and alignment across modalities, we introduce a supervised contrastive loss between paired speech-text representations and a KL-divergence-based regulariza-tion across expert predictions. Importantly, MiSTER-E does not rely on speaker identity at any stage. Experiments on three benchmark datasets-IEMOCAP, MELD, and MOSI-show that our proposal achieves 70.9%, 69.5%, and 87.9% weighted F1-scores respectively, outperforming several baseline speech-text ERC systems. We also provide various ablations to highlight the contributions made in the proposed approach.
Engineering Breakdown
Plain English
This paper addresses emotion recognition in multi-turn conversations by proposing MiSTER-E, a Mixture-of-Experts framework that handles both speech and text modalities separately before intelligently combining them. The core problem is that conversations have temporal context (who said what when) and multiple information sources (audio, words, tone) that traditional models struggle to integrate effectively. MiSTER-E uses fine-tuned LLMs to extract rich embeddings from speech and text independently, then applies a learned gating mechanism to dynamically weight predictions from three specialized experts: speech-only, text-only, and cross-modal. This modular approach decouples the challenges of understanding each modality from the challenge of fusing them together.
Core Technical Contribution
The key innovation is decoupling multimodal emotion recognition into two distinct problems: modality-specific context modeling (handled by LLM-based experts) and intelligent information fusion (handled by a learned gating mechanism). Rather than forcing a single model to simultaneously learn both speech/text representations and their optimal combination, MiSTER-E creates specialist experts for each modality and a separate routing layer that learns which expert to trust for each prediction. This is different from prior work that typically concatenates modality embeddings or uses early fusion strategies; MiSTER-E implements late fusion with learned expert weighting. The use of fine-tuned LLMs as the backbone for utterance embeddings, rather than standard transformers, provides richer contextual understanding of dialogue flow.
How It Works
The pipeline works as follows: for each utterance in a conversation, the system independently extracts embeddings using two fine-tuned LLM pathways—one optimized for speech transcripts and one for the raw text. These utterance-level embeddings then flow into a convolutional-recurrent context modeling layer that captures temporal dependencies across the conversation turn sequence (CNN captures local patterns, RNN captures long-range dialogue flow). Three expert heads make independent predictions: a speech-only expert uses speech embeddings, a text-only expert uses text embeddings, and a cross-modal expert combines both. Finally, a learned gating mechanism (likely a small neural network taking all embeddings as input) produces normalized weights for each expert's prediction, outputting a weighted ensemble decision. The entire system is trained end-to-end so the gating weights adapt to learn which modality or combination is most reliable for each dialogue context.
Production Impact
For production systems, MiSTER-E offers concrete advantages: the modular expert design allows you to independently debug, retrain, or swap out the speech or text pathways without affecting the entire system, reducing engineering complexity when one modality performs worse than expected. The learned gating mechanism provides explainability—you can inspect which modality the system trusted for each prediction, useful for understanding failure modes in customer interactions. However, this comes with significant compute overhead: you're essentially running three parallel inference paths plus the LLM-based embedding extraction, likely increasing latency by 2-3x compared to a single-modality baseline. For production call centers or conversational AI, you'd need to decide if the accuracy gain from multimodal fusion justifies the additional computational cost, and whether you have labeled conversation data to fine-tune the LLMs for your specific domain (emotionally diverse conversations are required for training).
Limitations and When Not to Use This
The paper assumes you have access to both high-quality speech and text transcripts for each utterance, which means dependency on accurate speech-to-text (a known source of errors in noisy environments). The approach likely requires substantial labeled data to fine-tune the LLMs and train the gating mechanism—it's unclear how the system performs in low-data regimes common in specialized domains like medical or legal conversations. The temporal context modeling uses CNN-RNN layers which may not scale well to very long conversations (100+ turns) or conversations with multiple concurrent speakers; no analysis of these edge cases is provided. The paper doesn't discuss how the approach handles missing modalities (e.g., video calls where audio fails, or text-only chat) or degraded speech quality, which are common in real deployment scenarios.
Research Context
This work builds on the established field of Emotion Recognition in Conversations (ERC), which has been studied on benchmarks like IEMOCAP and MELD datasets. Prior work in ERC typically used CNN-RNN architectures or graph neural networks to model dialogue structure, but rarely leveraged the representational power of modern LLMs or explicitly modeled expert specialization. The Mixture-of-Experts paradigm itself is well-established in scaling large models (see Switch Transformers, GLaM), but applying it to the fusion problem in multimodal emotion recognition is relatively novel. This work opens a research direction around modular multimodal architectures where different experts specialize in different information sources, which could extend beyond emotion recognition to tasks like sarcasm detection, intent recognition, or conversational understanding in general.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::