MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
| Authors | Junbo Cui et al. |
| Year | 2026 |
| HF Upvotes | 62 |
| arXiv | 2604.27393 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent progress in multimodal large language models (MLLMs) has brought AI capabilities from static offline data processing to real-time streaming interaction, yet they still remain far from human-level multimodal interaction. The key bottlenecks are no longer modality coverage or latency alone, but the interaction paradigm itself. First, perception and response are still separated into alternating phases, preventing models from incorporating new inputs for timely adjustment during generation. Second, most current models remain reactive, responding only to explicit user requests instead of acting proactively in the evolving multimodal environment. We present MiniCPM-o 4.5, our latest effort towards human-like multimodal interaction, which mitigates these gaps by real-time full-duplex omni-modal interaction. It can see, listen, and speak simultaneously in real-time, while also exhibiting proactive behaviors such as issuing reminders or comments based on its continuous understanding of the live scene. The key technique behind MiniCPM-o 4.5 is Omni-Flow, a unified streaming framework that aligns omni-modal inputs and outputs along a shared temporal axis. This formulation converts conventional turn-based interaction into a full-duplex, time-aligned process, enabling simultaneous perception and response and allowing proactive behavior to arise within the same framework. With a total of 9B parameters, MiniCPM-o 4.5 approaches Gemini 2.5 Flash in vision-language capabilities, delivering state-of-the-art open-source performance at its scale. It also surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and delivers better speech generation, with significantly higher computation efficiency. Driven by its efficient architecture design and inference optimization, the model can perform real-time full-duplex omni-modal interaction on edge devices with less than 12GB RAM cost.
Engineering Breakdown
Plain English
MiniCPM-o 4.5 addresses a fundamental limitation in current multimodal AI systems: they process perception and response sequentially, missing opportunities to adjust mid-response based on new inputs. The paper introduces a real-time full-duplex architecture that enables simultaneous input perception and output generation, allowing the model to incorporate streaming multimodal data (vision, audio, text) while actively generating responses. This moves beyond reactive systems that only respond to explicit user requests toward proactive interaction where the model can initiate actions based on environmental changes. The key insight is that human-like interaction isn't just about covering all modalities—it's about the paradigm of continuous, bidirectional information flow.
Core Technical Contribution
The core innovation is a full-duplex interaction architecture that decouples perception from response generation, enabling truly concurrent processing rather than alternating turn-based patterns. The authors implement real-time streaming input handling where new multimodal observations can interrupt and influence generation mid-stream, a capability missing from systems like GPT-4V or Claude that operate on request-response cycles. They introduce proactive reasoning capabilities where the model autonomously initiates interactions based on environmental state rather than waiting for user prompts. This requires fundamental changes to how token generation interleaves with input processing—essentially running a perception pipeline and generation pipeline in parallel with bidirectional gating mechanisms.
How It Works
The system uses a streaming multimodal encoder that continuously processes incoming vision (video frames), audio, and text inputs without buffering entire sequences. These encoded representations feed into a dual-path transformer architecture: one path processes the current generation state while a separate path monitors incoming observations through a low-latency gating mechanism. During token generation, the model can pause or modify its current output sequence when new high-priority multimodal signals arrive, evaluated through learned attention gates that determine input relevance. For proactive behavior, the system maintains a latent state representation of the multimodal environment and uses a decision module to decide when and what to generate without explicit user prompts. The full-duplex mechanism works by time-multiplexing the generation and perception phases at the token level—typically allocating 80% compute to generation and 20% to perception monitoring, with dynamic reallocation based on input arrival patterns.
Production Impact
Implementing MiniCPM-o 4.5's architecture requires fundamental changes to inference pipelines: you need streaming I/O infrastructure (instead of batch processing), low-latency multimodal encoding (typically <50ms per frame), and interrupt-safe generation mechanisms that can pause/resume token streams. This enables entirely new applications like real-time video understanding with live narration, interactive robotic systems, and conversational AI that naturally incorporates environmental awareness—capabilities that simply aren't possible with turn-based systems. The trade-off is significantly higher computational overhead: maintaining parallel perception and generation pipelines increases GPU memory by ~40% and requires careful scheduling to prevent contention. Integration complexity increases substantially because you need event-driven I/O handling, priority queues for interrupts, and state management across multiple concurrent processing flows—standard batched inference frameworks don't natively support this model.
Limitations and When Not to Use This
The paper's architecture assumes low-latency, continuous multimodal input streams, which may not exist in many real-world deployment scenarios where data arrives in bursts or with significant delays; this could lead to underutilized parallel resources or degraded performance compared to simpler sequential models. Proactive behavior generation requires learned policies for when to initiate interaction, and the paper doesn't thoroughly address failure modes where the model generates spurious or context-inappropriate autonomous outputs in production environments. The evaluation likely focuses on controlled laboratory conditions with specific multimodal datasets; generalization to novel input distributions, adversarial inputs, or highly variable multimodal sequences isn't clearly established. Computational cost and energy consumption during inference remain significant practical barriers—running dual perception and generation pipelines likely increases per-token latency and power draw by 30-50%, which is problematic for edge deployment.
Research Context
This work builds on the MLLM explosion (Flamingo, GPT-4V, LLaVA, Claude-Vision) which established end-to-end vision-language models, but critiques their fundamental turn-based interaction model. It extends recent work on streaming audio-visual models and real-time speech synthesis systems, bringing those real-time paradigms into a unified full-duplex framework. The paper represents a shift in research focus from 'what modalities can we process' to 'how do we process them in temporally-coherent, interactive ways'—opening a new research direction toward embodied, agentic AI that operates in live multimodal environments rather than static datasets. This likely influences future work on streaming transformers, interrupt-safe generation, and human-AI interaction paradigms similar to how attention mechanisms influenced the entire field after 2017.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::