Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video
| Authors | Chanhyuk Choi et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.07786 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Talking face generation has gained significant attention as a core application of generative models. To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role. However, existing approaches often limit expressive flexibility and struggle to generate extended emotions. Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions. Audio-based methods can leverage emotionally rich speech signals - and even benefit from expressive text-to-speech (TTS) synthesis - but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches. Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (e.g., sarcasm). To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces. C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities. Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos - even for unseen extended emotions. Code, checkpoint, and demo are available at https://chanhyeok-choi.github.io/C-MET/
Engineering Breakdown
Plain English
This paper addresses emotion editing in talking face videos—a key problem for making synthesized talking heads more expressive and realistic. Existing methods suffer from three fundamental limitations: label-based approaches use discrete emotion categories that can't capture nuanced feelings; audio-based methods struggle because emotion and linguistic content are entangled in emotional speech; and image-based methods require manually selected reference images. The authors propose a cross-modal emotion transfer approach that decouples emotion from content, enabling flexible emotion editing with extended emotional sequences that go beyond what prior methods could generate.
Core Technical Contribution
The core novelty is a cross-modal framework that separates emotion representation from linguistic content in talking face generation. Rather than relying on discrete labels, audio features, or reference images alone, the method learns to transfer emotional characteristics across modalities while keeping the linguistic content of the speech intact. This is achieved through a mechanism that extracts emotion-specific features independent of the actual speech content, allowing fine-grained control over which emotions to apply to any given speech sequence. The key insight is that emotions can be represented and transferred as continuous, modality-agnostic features rather than being tied to specific audio signals or reference identities.
How It Works
The system takes three main inputs: a source talking face video or audio, target emotion specifications (which can come from reference images, text descriptions, or continuous emotion vectors), and the linguistic content to be preserved. The architecture first disentangles emotion features from linguistic content through a dedicated encoder that learns emotion-invariant speech representations. These emotion features are then extracted from the target (either by analyzing reference images or through other modalities) and combined with the preserved linguistic content. A decoder then generates the final talking face video frame-by-frame, where each frame's expression and lip-sync are guided by the blended emotion-linguistic representation. The cross-modal aspect means the system can accept emotion cues from different sources (images, text, audio) and apply them consistently to any speech input.
Production Impact
For teams building talking head synthesis systems, this approach eliminates the need for manual reference image selection—a major bottleneck in current production pipelines where creating emotionally consistent content requires hand-curating reference frames. You could train once on a diverse emotion dataset and then apply any emotion to any speech input at inference time, dramatically improving scalability for content creation platforms. The trade-off is additional computational overhead during training to learn the cross-modal disentanglement, likely requiring 1.5-2x the training time and data compared to single-modality baselines. Inference latency impact should be minimal since the decoupling happens offline, but you'd need to maintain emotion encoders for multiple modalities, increasing model size and complexity. This is particularly valuable for personalized video generation, dubbing workflows, and accessibility applications where you need fine emotional control independent of source material.
Limitations and When Not to Use This
The paper abstract doesn't fully specify how well the method handles extreme or mixed emotions—discrete emotion categories were a limitation, but continuous emotion vectors may suffer from poorly-defined emotion space boundaries. The approach assumes clean separation between emotion and linguistic content is possible, which may not hold for inherently emotional speech acts like sarcasm or irony where tone and meaning are inseparable. Training data requirements are likely substantial since you need aligned samples across multiple modalities with consistent emotion labels, making this approach less practical for low-resource languages or specialized domains. The paper doesn't address temporal consistency—whether the generated emotions maintain coherent trajectories over long sequences or if frame-to-frame jitter occurs, which is critical for extended videos.
Research Context
This work builds on recent advances in talking face generation using neural rendering and generative models, addressing a known weakness: the lack of fine-grained emotion control in synthesized talking heads. It extends prior work on disentangled representation learning (where emotion, identity, and content are separated) into the specific domain of cross-modal transfer. The contribution directly challenges label-based emotion models by showing that continuous, modality-agnostic representations are more expressive. This opens research directions in other cross-modal generation tasks (e.g., emotion transfer in speech synthesis, gesture generation) and could influence how emotion is represented in multimodal AI systems more broadly.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::