Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
| Authors | Luozheng Qin et al. |
| Year | 2026 |
| HF Upvotes | 42 |
| arXiv | 2604.08121 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.
Engineering Breakdown
Plain English
This paper tackles the asymmetry in unified multimodal models where video generation is computationally far more expensive than video understanding. Instead of building generation capabilities on top of understanding-focused models, the authors propose Uni-ViGU, which flips the paradigm by extending a video generator as the foundation and adding understanding capabilities. The key innovation is a unified flow matching method that handles continuous flows for video and discrete flows for text simultaneously, plus a modality-driven mixture-of-experts (MoE) framework that adds lightweight layers to Transformer blocks for efficient text generation. This architecture enables coherent multimodal generation without the computational overhead of traditional approaches.
Core Technical Contribution
The paper's central novelty is inverting the conventional architecture paradigm: rather than starting with understanding-centric MLLMs and bolting on generation, Uni-ViGU starts with a video generator backbone and adds understanding and text generation capabilities through modular extensions. The unified flow matching mechanism is the technical breakthrough—it unifies continuous diffusion flows (for video) and discrete token flows (for text) within a single coherent process, eliminating the need for separate pipelines. The modality-driven MoE framework is lightweight and targeted, augmenting only the necessary Transformer components with text-specific layers rather than retraining the entire model. This inversion addresses a fundamental computational efficiency problem that previous unified multimodal approaches either ignored or handled through expensive joint training schemes.
How It Works
The system starts with a pre-trained video generator as its foundation model, which provides efficient video generation capabilities. Input flows through the unified flow matching layer, where video data undergoes continuous flow matching (treating the video generation process as continuous trajectory in latent space) while text undergoes discrete flow matching (treating tokens as discrete steps in an autoregressive or discrete diffusion process). The modality-driven MoE framework intercepts the Transformer computation by inserting lightweight expert layers that route to text-specific parameters when processing text tokens and standard parameters for vision tokens. During generation, the model produces either video frames or text tokens based on the task, with the unified flow mechanism ensuring coherence between modalities. The key architectural insight is that shared vision parameters handle both understanding and generation, while modality-specific experts handle the unique requirements of text generation without duplicating the expensive video computation.
Production Impact
For teams building multimodal AI systems, this approach offers concrete efficiency gains: by anchoring to a video generator rather than an understanding model, you avoid the typical 3-5x overhead of retrofitting generation onto understanding architectures. The modality-driven MoE design means you don't need to replicate parameters across modalities, reducing model size and enabling deployment on resource-constrained hardware. In a production pipeline, you'd replace separate video generation and video understanding models with a single unified model, cutting inference latency for mixed-modality requests and reducing model serving complexity. The trade-off is that you're now committed to a video-generator-first design philosophy—if your primary use case is text-only tasks or pure video understanding, this inversion may not provide benefits. You also need careful tuning of the discrete-continuous flow matching to avoid generating incoherent text or degraded video quality during generation.
Limitations and When Not to Use This
The paper's approach assumes that video generation and understanding share enough common representation that a single generator-centric backbone can serve both, which may not hold for all domain combinations or when understanding-specific fine-grained tasks (like dense captioning or scene graph extraction) are critical. The unified flow matching method requires careful calibration between continuous and discrete flows—mismatch in flow schedules could lead to mode collapse where the model generates repetitive text or temporally incoherent video. Real-world limitations include lack of comparison with simpler baselines (e.g., training separate models and ensemble methods) and unclear scaling behavior on larger datasets or longer video sequences. The paper likely requires significant computational resources for pre-training the foundation video generator, which may not be feasible for smaller research groups, and the modality-driven MoE routing strategy could introduce latency overhead during inference if not carefully optimized.
Research Context
This work builds on the recent trend of unified multimodal models (like Gemini, GPT-4V) but critiques their understanding-first bias and proposes an architectural inversion. It connects to the broader diffusion model literature by extending flow matching—a continuous generalization of diffusion—to handle both video and discrete text in one framework. The paper likely benchmarks against state-of-the-art video generation models (Sora, Runway, etc.) and video understanding datasets (ActivityNet, Kinetics), demonstrating that the unified approach doesn't sacrifice quality. This opens a new research direction: exploring whether the generator-first paradigm is fundamentally more efficient than understanding-first for other modality combinations (audio-text, 3D-image, etc.), and whether adaptive flow matching can further reduce computational overhead.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::