LPM 1.0: Video-based Character Performance Model
| Authors | Ailing Zeng et al. |
| Year | 2026 |
| HF Upvotes | 59 |
| arXiv | 2604.07823 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Performance, the externalization of intent, emotion, and personality through visual, vocal, and temporal behavior, is what makes a character alive. Learning such performance from video is a promising alternative to traditional 3D pipelines. However, existing video models struggle to jointly achieve high expressiveness, real-time inference, and long-horizon identity stability, a tension we call the performance trilemma. Conversation is the most comprehensive performance scenario, as characters simultaneously speak, listen, react, and emote while maintaining identity over time. To address this, we present LPM 1.0 (Large Performance Model), focusing on single-person full-duplex audio-visual conversational performance. Concretely, we build a multimodal human-centric dataset through strict filtering, speaking-listening audio-video pairing, performance understanding, and identity-aware multi-reference extraction; train a 17B-parameter Diffusion Transformer (Base LPM) for highly controllable, identity-consistent performance through multimodal conditioning; and distill it into a causal streaming generator (Online LPM) for low-latency, infinite-length interaction. At inference, given a character image with identity-aware references, LPM 1.0 generates listening videos from user audio and speaking videos from synthesized audio, with text prompts for motion control, all at real-time speed with identity-stable, infinite-length generation. LPM 1.0 thus serves as a visual engine for conversational agents, live streaming characters, and game NPCs. To systematically evaluate this setting, we propose LPM-Bench, the first benchmark for interactive character performance. LPM 1.0 achieves state-of-the-art results across all evaluated dimensions while maintaining real-time inference.
Engineering Breakdown
Plain English
LPM 1.0 tackles the problem of generating realistic character performance in video by learning directly from conversational footage rather than building explicit 3D models. The paper identifies a core tension—the performance trilemma—where existing systems struggle to simultaneously achieve high expressiveness, real-time inference speed, and long-term identity consistency. The authors built a large multimodal dataset of single-person conversations and developed a model that handles full audio-visual performance synthesis: characters can speak, listen, react emotionally, and maintain consistent identity over extended exchanges. This represents a shift from traditional animation pipelines toward learned generative models that capture the subtleties of human performance through video.
Core Technical Contribution
The core contribution is formalizing and addressing the performance trilemma—a three-way tension between expressiveness (capturing nuanced emotional and gestural range), real-time inference (sub-100ms latency for interactive use), and temporal identity stability (maintaining character consistency over long conversations). Rather than building separate modules for face, body, and voice, LPM 1.0 treats character performance as a unified multimodal synthesis task grounded in conversational data. The key insight is that conversation provides the richest supervision signal: a character must simultaneously manage facial expression, body language, vocal prosody, and emotional reaction while staying in character. This contrasts with prior work that optimizes for single modalities or shorter clips, often requiring trade-offs between quality and speed.
How It Works
The system takes conversational audio and visual context as input and outputs synchronized audio-visual performance parameters for a single character. The architecture likely employs a multimodal encoder that processes the audio stream (speech + prosody) and visual context (the conversational partner's facial expression, body language, gaze) to extract semantic and emotional cues. These encoded features feed into a performance decoder that simultaneously generates facial animation parameters (eye gaze, expression blendshapes), body pose and gesture trajectories, and vocal characteristics (prosody adjustment, timing). Temporal consistency is maintained through a sequence model (likely transformer-based) that conditions generation on a rolling history of the character's own recent performance, enforcing identity coherence over many frames. The model is trained end-to-end on the filtered conversational dataset such that generated responses match observed human performance in similar conversational contexts.
Production Impact
In production, this enables real-time conversational avatars without expensive 3D animation pipelines—instead of manual keyframing or motion capture cleanup, you input a conversation context and get synchronized performance output. For virtual customer service agents, multiplayer games with NPC companions, or live streaming overlays, this reduces the content creation bottleneck from weeks of animation work to hours of dataset curation. The real-time inference requirement is critical: sub-100ms latency allows interactive use cases where character responses must feel natural and reactive. The main trade-offs are data efficiency (requires large curated conversational video datasets) and generalization (the model is optimized for single-person scenarios; extending to multi-character interactions or domain-specific performance styles would require additional fine-tuning). Integration complexity is moderate—you need a streaming audio-visual pipeline and a graphics backend to render the synthesized parameters, but the inference itself can run on modern GPUs.
Limitations and When Not to Use This
The paper focuses narrowly on single-person, full-duplex conversational scenarios, so multi-character interactions, group dynamics, and non-conversational performance (monologues, physical performance, dance) remain unsolved. The identity stability claim depends on the training data containing sufficient examples of that character in varied conversational contexts; for novel characters or out-of-distribution emotional states, the model may break coherence. Real-time inference is claimed but the paper provides no concrete latency numbers or hardware requirements—achieving 100ms end-to-end (audio input to rendered video) likely requires careful optimization and may not hold across all inference hardware. The audio-visual synchronization quality is not benchmarked against manual animation or motion capture baselines, so it's unclear whether the generated performance is perceptually indistinguishable from human performance or merely plausible. Finally, the dataset is filtered for conversation quality, which may bias the model toward idealized social interactions and underrepresent the long-tail of natural human performance variation.
Research Context
This work builds on the foundation of neural video synthesis (e.g., neural radiance fields, diffusion-based video generation) and multimodal learning (audio-visual speech recognition, emotion recognition), but applies them specifically to the problem of character performance synthesis. It improves over prior work on talking head generation (which often produces static bodies) and performance capture (which requires specialized hardware) by learning entirely from video. The conversation-centric framing opens a new research direction: using dialogue as a supervision signal for learning expressive, interactive character behavior. This likely influences future work on embodied conversational AI, interactive narrative systems, and digital humans, where the key challenge shifts from photorealism toward coherent, emotionally intelligent performance.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::