Mode Seeking meets Mean Seeking for Fast Long Video Generation
| Authors | Shengqu Cai et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24289 |
| Download | |
| Categories | cs.CV, cs.LG |
Abstract
Scaling video generation from seconds to minutes faces a critical bottleneck: while short-video data is abundant and high-fidelity, coherent long-form data is scarce and limited to narrow domains. To address this, we propose a training paradigm where Mode Seeking meets Mean Seeking, decoupling local fidelity from long-term coherence based on a unified representation via a Decoupled Diffusion Transformer. Our approach utilizes a global Flow Matching head trained via supervised learning on long videos to capture narrative structure, while simultaneously employing a local Distribution Matching head that aligns sliding windows to a frozen short-video teacher via a mode-seeking reverse-KL divergence. This strategy enables the synthesis of minute-scale videos that learns long-range coherence and motions from limited long videos via supervised flow matching, while inheriting local realism by aligning every sliding-window segment of the student to a frozen short-video teacher, resulting in a few-step fast long video generator. Evaluations show that our method effectively closes the fidelity-horizon gap by jointly improving local sharpness, motion and long-range consistency. Project website: https://primecai.github.io/mmm/.
Engineering Breakdown
Plain English
This paper tackles the problem of generating long-form videos (minute-scale) by proposing a novel training paradigm that separates local visual fidelity from long-term narrative coherence. The authors use a Decoupled Diffusion Transformer with two distinct heads: a global Flow Matching head that learns long-video structure via supervised learning, and a local Distribution Matching head that maintains short-video quality by aligning with a frozen teacher model using mode-seeking reverse-KL divergence. This decoupled approach addresses a key bottleneck in video generation—abundant short-video training data is high-quality but lacks long-form coherence, while long-video data is scarce and domain-limited. The method enables synthesis of minute-scale videos that maintain both local fidelity and global narrative structure without requiring large-scale long-video datasets.
Core Technical Contribution
The core novelty is the decoupling of local fidelity optimization from long-term coherence learning through a unified Diffusion Transformer representation. Rather than trying to train a single model end-to-end on scarce long-video data, the authors leverage asymmetric data availability: abundant short videos train the local quality head via mode-seeking (reverse-KL divergence to a frozen teacher), while limited long videos supervise the global structure head via Flow Matching. This is fundamentally different from prior video generation work that either trains on short clips independently or attempts joint training on mixed-quality data. The mode-seeking approach (reverse-KL instead of forward-KL) is technically important because it prevents the local head from collapsing to a single mode and maintains diversity while staying faithful to the short-video teacher distribution.
How It Works
The system ingests video sequences and processes them through a shared Diffusion Transformer backbone that branches into two specialized heads. The global Flow Matching head operates on full long-video sequences and is trained with supervised learning objectives on the limited long-video data available, learning to predict frame-to-frame optical flow or motion patterns that encode narrative structure and temporal coherence. The local Distribution Matching head processes sliding windows of frames and uses reverse-KL divergence (mode-seeking) to align its output distribution with a frozen short-video teacher model, ensuring each local chunk maintains the visual quality standards established by high-quality short-video data. During generation, both heads contribute to the diffusion sampling process: the global head provides long-range guidance while the local head ensures pixel-level quality. The frozen teacher prevents mode collapse and distribution drift in the local head without requiring continuous retraining on massive short-video datasets.
Production Impact
For production video generation systems, this approach would dramatically reduce data requirements by decoupling the expensive long-video annotation problem from short-video quality optimization. Teams could deploy this by: (1) using existing short-video datasets to train and freeze the local head once, then (2) collecting or licensing smaller targeted long-video datasets for specific domains (e.g., cinematic storytelling, sports highlights), and (3) fine-tuning only the global Flow Matching head on domain data. This significantly lowers the barrier to entry for building long-form video generation products compared to approaches requiring massive balanced datasets. The trade-off is architectural complexity—managing two heads, maintaining teacher-student alignment, and potentially higher inference compute due to sliding window processing and dual optimization paths. Latency impact depends on video length and sliding window size, but the frozen teacher approach reduces per-generation retraining overhead compared to online distillation methods.
Limitations and When Not to Use This
The paper assumes sufficient long-video data exists within target domains for the global head to learn coherent structure—this breaks down for truly novel content types or when domain-specific long videos are unavailable. The frozen teacher model constrains the local head to historical short-video quality distributions and may prevent learning novel visual styles that weren't well-represented in training data. Temporal consistency across minute-scale videos isn't explicitly addressed in the abstract; maintaining character/object identity over 60+ seconds of generation remains an open challenge that mode-seeking alone may not solve. The approach requires careful hyperparameter tuning of the mode-seeking reverse-KL weighting and sliding window overlap, making it less plug-and-play than end-to-end trained models. Additionally, the paper doesn't discuss failure modes when long-video and short-video distributions are significantly misaligned (e.g., short videos emphasize aesthetic detail while long videos emphasize motion), which could create artifacts at composition boundaries.
Research Context
This work builds on recent advances in diffusion transformers and Flow Matching as alternatives to autoregressive generation, while extending ideas from distillation and mode-seeking divergences used in generative modeling. It directly addresses limitations identified in prior long-video generation research (e.g., Runway's Gen-3, Pika) that either struggled with coherence or required massive balanced datasets. The decoupling paradigm is conceptually related to curriculum learning and multi-task learning but applied specifically to the video domain where data scarcity is asymmetric—abundant short videos, scarce long videos. This opens a research direction toward more practical, data-efficient video generation by designing architectures that exploit natural data asymmetries rather than requiring balanced large-scale datasets.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::