Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
| Authors | Lijiang Li et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2603.06577 |
| Download | |
| Categories | cs.CV |
Abstract
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering research, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.
Engineering Breakdown
Plain English
This paper introduces Omni-Diffusion, a multimodal language model that replaces the standard autoregressive transformer backbone with mask-based discrete diffusion models to handle text, speech, and image understanding and generation in a unified framework. While recent multimodal LLMs have achieved strong results, they're limited by their sequential autoregressive design which is computationally expensive and may not be optimal for all modalities. The authors' key insight is that discrete diffusion models—which have shown promise in vision and generation tasks separately—can serve as a flexible, efficient backbone that handles any-to-any multimodal tasks without specializing architectures for different modalities. This represents a fundamental rethinking of multimodal architecture design beyond the dominant transformer paradigm.
Core Technical Contribution
The core novelty is applying mask-based discrete diffusion as the unified backbone for a truly any-to-any multimodal system, rather than using diffusion as a component within a transformer or using separate diffusion models for generation tasks. Unlike autoregressive models that generate tokens sequentially (which is inherently sequential and forces left-to-right dependencies), diffusion models iteratively refine a noisy representation across parallel positions, enabling bidirectional context and more flexible generation patterns. This architectural choice enables Omni-Diffusion to natively handle multiple modalities (text, speech, images) with the same core mechanism, eliminating the need for modality-specific adapters or specialized decoding strategies that plague current MLLMs. The discrete diffusion formulation—working directly on token/quantized representations rather than continuous pixel spaces—is specifically suited for language-like discrete outputs while remaining competitive with continuous diffusion for images.
How It Works
Omni-Diffusion takes multimodal inputs (text tokens, speech spectrograms, image patches) and converts them to a unified discrete token representation using shared quantization codebooks. The diffusion process operates by starting with a fully masked input sequence, where each position is initially unknown, and iteratively denoises across all positions in parallel over T diffusion steps. At each step, the model conditions on unmasked tokens (those already refined) and predicts which positions to unmask next, gradually filling in the full multimodal sequence. The key architectural difference from autoregressive models is that there's no fixed left-to-right generation order—the model learns which positions are most confident and unmasks high-confidence positions first, enabling flexible, context-aware generation. During training, the model learns to predict token identities at masked positions given the context of unmasked positions, effectively learning bidirectional multimodal representations. Inference can be directed toward generation tasks (progressively unmask all positions starting from user-provided prompts) or understanding tasks (unmask only the target portion conditioned on given context).
Production Impact
This approach could fundamentally simplify multimodal systems by eliminating separate pipelines for understanding versus generation—a single Omni-Diffusion checkpoint handles both, reducing model inventory and deployment complexity. Engineers currently building multimodal systems maintain separate models or complex branching logic to handle generation (decoder models or diffusion pipelines) versus understanding (encoder-decoder transformers). The parallel denoising nature of diffusion could reduce latency compared to autoregressive generation once amortized over the fixed number of diffusion steps, though it introduces a different computational profile: total FLOPs may be higher but with better parallelization and potentially lower peak memory. Integration with existing inference infrastructure would require adapting sampling and batching logic since diffusion sampling differs fundamentally from autoregressive token prediction—this has non-trivial implications for serving systems built around batched token generation. The unified architecture would reduce engineering burden: model serving, fine-tuning, and prompt engineering become more uniform across tasks, though practitioners would need to relearn generation patterns (e.g., controlling generation quality via noise schedules rather than temperature/top-k).
Limitations and When Not to Use This
The paper doesn't demonstrate concrete speed or quality comparisons against state-of-the-art MLLMs like GPT-4V or recent diffusion-based vision models, making it unclear whether the architectural elegance translates to practical advantages—diffusion's fixed number of steps might make it slower than modern autoregressive models that can generate fast with early stopping. Discrete diffusion on language has seen limited adoption compared to continuous diffusion for images, so the scalability to very large token vocabularies and language understanding tasks is unproven; language modelers have historically preferred autoregressive or bidirectional transformer designs for good reasons. The approach assumes a unified token representation works across modalities, but vision and language have fundamentally different statistical properties (images are locally coherent, language is highly sequential with long-range dependencies), and it's unclear how well a single diffusion process balances these. The paper is incomplete in the abstract and doesn't provide experimental results, so claims about efficiency or quality are currently unvalidated—this could be addressing a theoretical question rather than a practical engineering problem.
Research Context
This work builds on two recent research threads: (1) the success of discrete diffusion models in vision tasks and image generation, showing diffusion is competitive beyond continuous image synthesis, and (2) the dominance of autoregressive transformers in language and multimodal modeling, which has prompted researchers to explore alternatives like non-autoregressive and bidirectional architectures. It extends prior work applying diffusion to discrete domains (e.g., discrete diffusion for text generation, diffusion for image understanding) by proposing a unified system that combines these ideas. The research direction aligns with a broader push toward more efficient and flexible multimodal architectures, responding to the computational costs of scaling current MLLMs and the inherent limitations of autoregressive decoding for parallelization. Future work in this direction could explore hybrid architectures mixing diffusion and autoregression, or applying this framework to other modalities like video or 3D data.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::