RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 9 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Yanzuo Lu et al. |
| Year | 2026 |
| HF Upvotes | 9 |
| arXiv | 2605.15190 |
| Download | |
| Code | https://github.com/mvp-ai-lab/RAVEN |
Abstract
Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content. Distilling such generators from high-fidelity bidirectional teachers yields competitive few-step models, yet a persistent gap between the history distributions encountered during training and those arising at inference constrains generation quality over long horizons. We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states. This formulation aligns training attention with inference-time extrapolation and allows downstream chunk losses to supervise the history representations on which future predictions depend. We further propose Consistency-model Group Relative Policy Optimization (CM-GRPO), which reformulates a consistency sampling step as a conditional Gaussian transition and applies online Reinforcement Learning (RL) directly to this kernel, avoiding the Euler-Maruyama auxiliary process adopted in prior flow-model RL formulations. Experiments demonstrate that RAVEN surpasses recent causal video distillation baselines across quality, semantic, and dynamic degree evaluations, and that CM-GRPO provides further gains when combined with RAVEN.
Engineering Breakdown
Plain English
RAVEN tackles a real problem in streaming video generation: autoregressive diffusion models that generate video frame-by-frame work well during training but degrade over long sequences because the model encounters different data distributions at inference than it saw during training. The paper introduces a training technique that simulates inference conditions by mixing clean historical content with noisy intermediate states, allowing the model to learn robust representations that hold up across many generation steps.
Key Engineering Insight
The core insight is that training-time rollouts should explicitly mirror inference conditions—by repacking generated content as alternating clean endpoints and noisy denoising states, you supervise the model on the exact distribution mismatch it'll face at inference, eliminating the classic train-test gap that plagues sequential generation.
Why It Matters for Engineers
For engineers building real-time video generation systems, long-horizon quality degradation is a showstopper—users notice flickering or incoherence after a few seconds. This approach directly addresses that by training models that stay stable over extended generation windows, making streaming video systems practical without massive compute overhead for inference-time corrections.
Research Context
Prior work on distilled autoregressive video diffusion achieved few-step inference but suffered from compounding errors over time due to distribution shift. RAVEN advances this by making the training process aware of how autoregressive rollouts accumulate error, enabling models to maintain fidelity at longer horizons. This bridges the gap between efficient real-time generation and quality, unlocking practical deployment of video extrapolation in production systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::