Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
| Authors | Zeyue Tian et al. |
| Year | 2026 |
| HF Upvotes | 40 |
| arXiv | 2604.10708 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.
Engineering Breakdown
Plain English
Audio-Omni is the first unified framework that handles audio understanding, generation, and editing across speech, music, and general sound domains in a single end-to-end system. Previous approaches required separate specialized models for each capability and domain, limiting flexibility and forcing engineers to stitch together multiple pipelines. The system combines a frozen Multimodal Large Language Model for reasoning with a trainable Diffusion Transformer for high-fidelity synthesis, enabling seamless multi-task audio processing without domain-specific constraints.
Core Technical Contribution
The core novelty is unifying three traditionally separate audio tasks—understanding, generation, and editing—into a single trainable architecture that works across multiple domains without task-specific or domain-specific fine-tuning. Rather than building separate models for speech synthesis, music generation, and sound editing, the authors designed a shared token-based representation space where the frozen MLLM provides semantic understanding and the diffusion transformer handles the actual audio synthesis. The key architectural insight is that high-level multimodal reasoning (via the MLLM) can be decoupled from low-level audio quality (via the diffusion transformer), allowing the same framework to handle generation and editing tasks through a unified conditioning mechanism.
How It Works
The system takes multimodal inputs (text, audio, images) and passes them through a frozen Multimodal Large Language Model, which produces semantic tokens and understanding of the task intent. These tokens condition a trainable Diffusion Transformer that iteratively denoises audio latent representations, generating or editing audio waveforms in the target domain (speech, music, or general sound). The diffusion process works in latent space rather than raw waveforms, reducing computational cost while maintaining fidelity. For editing tasks, the system masks regions of audio and uses the same diffusion mechanism to in-paint or transform those regions, allowing speech style transfer, music arrangement changes, or sound effect editing without retraining. The architecture preserves the frozen MLLM's general understanding capabilities while the diffusion transformer learns domain-agnostic audio synthesis patterns during training, enabling zero-shot or few-shot transfer to new domains and tasks.
Production Impact
This approach eliminates the need for audio engineers to maintain separate pipelines for generation, editing, and understanding—a significant operational simplification for companies building multimodal applications. Instead of deploying three separate models (a speech synthesizer, music generator, and audio editor) with different inference latencies and memory requirements, you'd deploy a single unified system that handles all tasks. The main production trade-offs are: inference latency increases due to the iterative diffusion process (typically slower than auto-regressive models for generation), memory footprint grows because you must keep the frozen MLLM in memory alongside the diffusion transformer, and serving becomes complex because the system needs batching strategies optimized for variable-length diffusion steps. However, for applications requiring flexible audio editing and generation without task-specific retraining, this framework significantly reduces deployment and maintenance complexity compared to multi-model approaches.
Limitations and When Not to Use This
The paper doesn't fully address inference speed—diffusion-based generation is inherently iterative and slower than auto-regressive or flow-matching alternatives, making real-time editing challenging for latency-sensitive applications. The approach relies on a frozen MLLM, meaning performance is limited by that model's capabilities and biases; if the base MLLM struggles with certain audio descriptions or multimodal reasoning tasks, those weaknesses propagate through the framework. The abstract cuts off, but typically unified frameworks like this struggle with extreme quality on highly specialized tasks (e.g., professional music production-grade synthesis may require task-specific fine-tuning that outperforms the unified model). The system's generalization to truly out-of-distribution audio domains or novel synthesis tasks not well-represented in training data is unknown—unified frameworks often sacrifice peak performance in specific domains for broad average performance.
Research Context
Audio-Omni builds on recent advances in multimodal LLMs (like GPT-4V) extended to audio, and follows the paradigm of using frozen language models as semantic reasoners (similar to approaches in vision-language models and video understanding). It extends prior work like AudioLDM and Audiogen that addressed either generation or understanding in isolation, but attempts the first true unification across both generation and editing in a single framework. The research direction opens up compositional audio understanding—using language models to reason about what edits or generation are needed, then executing those operations through a shared diffusion backbone. This aligns with broader trends toward multimodal foundation models and task-agnostic synthesis, suggesting future work may combine this with other modalities (video, 3D) for truly end-to-end multimedia synthesis.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::