UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors
| Authors | Houyuan Chen et al. |
| Year | 2026 |
| HF Upvotes | 40 |
| arXiv | 2605.00658 |
| Download | |
| Code | https://github.com/houyuanchen111/UniVidX |
Abstract
Recent progress has shown that video diffusion models (VDMs) can be repurposed for diverse multimodal graphics tasks. However, existing methods often train separate models for each problem setting, which fixes the input-output mapping and limits the modeling of correlations across modalities. We present UniVidX, a unified multimodal framework that leverages VDM priors for versatile video generation. UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space, adapts to modality-specific distributions while preserving the backbone's native priors, and promotes cross-modal consistency during synthesis. It is built on three key designs. Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional conditional generation instead of fixed mappings. Decoupled Gated LoRA (DGL) introduces per-modality LoRAs that are activated when a modality serves as the generation target, preserving the strong priors of the VDM. Cross-Modal Self-Attention (CMSA) shares keys and values across modalities while keeping modality-specific queries, facilitating information exchange and inter-modal alignment. We instantiate UniVidX in two domains: UniVid-Intrinsic, for RGB videos and intrinsic maps including albedo, irradiance, and normal; and UniVid-Alpha, for blended RGB videos and their constituent RGBA layers. Experiments show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1,000 videos. Project page: https://houyuanchen111.github.io/UniVidX.github.io/
Engineering Breakdown
Plain English
UniVidX is a unified framework that repurposes video diffusion models (VDMs) to handle multiple different multimodal tasks—like image-to-video, depth-to-video, and sketch-to-video—without training separate models for each one. Instead of fixing input-output mappings, the system formulates all pixel-aligned tasks as conditional generation in a shared multimodal space, allowing it to learn correlations across different input modalities. The key innovation is Stochastic Condition Masking (SCM), which randomly partitions modalities into clean conditions and noisy targets during training, enabling the model to adapt to different modality distributions while preserving the pretrained backbone's native priors. This approach promotes cross-modal consistency during synthesis, meaning outputs remain coherent across different input types without retraining.
Core Technical Contribution
The core novelty is the unified multimodal framework that breaks away from the one-model-per-task paradigm by reformulating diverse pixel-aligned generation tasks as conditional generation in a shared space. Stochastic Condition Masking (SCM) is the algorithmic centerpiece—it randomly swaps which modalities are treated as clean conditions versus noisy targets during training, forcing the model to learn bidirectional mappings and modality-agnostic representations. This differs fundamentally from prior approaches that either train task-specific models or use simple concatenation of modalities; instead, UniVidX preserves the pretrained VDM backbone's generative priors while dynamically adapting to any input modality distribution. The framework also explicitly optimizes for cross-modal consistency through its synthesis process, ensuring that outputs respect correlations between different input types.
How It Works
UniVidX takes a pretrained video diffusion model backbone and extends it to handle multimodal conditional generation. During training, given a batch of examples with multiple modalities (e.g., RGB image, depth map, semantic mask), Stochastic Condition Masking randomly partitions these modalities—some become clean conditioning signals that guide generation, while others become noisy targets that the model must denoise during the reverse diffusion process. This random partitioning happens per-batch, forcing the model to learn robust representations that work regardless of which modality is the input. The architecture preserves the native VDM backbone (unchanged weights and structure) while inserting modality-specific adaptation layers that normalize each input modality's distribution before injection into the diffusion process. During inference, users specify which modalities are inputs (conditions) and which should be generated, and the model runs the standard diffusion reverse process conditioned on those inputs. Cross-modal consistency is maintained through the shared latent space and the learned alignments between modalities established during the SCM training procedure.
Production Impact
This approach dramatically reduces the engineering burden of deploying multimodal video generation systems. Instead of maintaining separate models for image-to-video, depth-to-video, sketch-to-video, and combinations thereof, teams can deploy a single unified model that handles all cases—reducing model serving infrastructure, GPU memory requirements, and deployment complexity. The framework is particularly valuable for applications requiring flexible input handling: video editing tools, 3D scene generation from multiple sensor inputs, autonomous driving simulation (RGB + LiDAR), or content creation pipelines that need to generate video from heterogeneous user inputs. The main trade-off is training cost: the SCM procedure requires more diverse data to adequately cover modality combinations, and the training time may be longer than task-specific models due to the bidirectional learning requirement. Latency during inference remains comparable to standard VDM inference since it uses the same diffusion backbone; the adaptation layers are lightweight and add minimal overhead.
Limitations and When Not to Use This
The paper's abstract is incomplete, cutting off mid-sentence after describing SCM, so we cannot assess the full scope of claims or results. UniVidX assumes all modalities can be meaningfully aligned in a shared space, which may not hold for modalities with fundamentally different semantics or scales (e.g., very high-dimensional lidar point clouds vs. low-dimensional categorical masks). The approach requires diverse training data covering multiple modality combinations, which can be expensive to collect and annotate—for niche applications, single-task models may still be more practical. The framework depends on a strong pretrained VDM backbone, so performance is ceiling-bound by the quality of that base model; it cannot fix fundamental limitations in the underlying diffusion architecture. Cross-modal consistency is promoted but not guaranteed, and failure modes around modality conflicts (e.g., inconsistent depth and RGB information) are not discussed in the abstract.
Research Context
This work builds on the recent trend of repurposing diffusion models beyond image generation into video and multimodal domains, extending prior work that showed VDMs could handle tasks like video inpainting or style transfer. It advances the field by addressing a key limitation of existing multimodal approaches: most either train task-specific models (limiting scalability) or use naive fusion strategies (losing modality-specific structure). The unified framework concept relates to broader research in foundation models and parameter-efficient adaptation—similar to how LoRA and other techniques allow single models to serve multiple tasks in NLP. This opens a research direction toward truly universal multimodal generative models that can flexibly consume any combination of input modalities, aligning with the trend toward more capable and adaptable foundation models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::