Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generation
| Authors | Xingtong Ge et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.03118 |
| Download | |
| Categories | cs.CV |
Abstract
Distilling video generation models to extremely low inference budgets (e.g., 2--4 NFEs) is crucial for real-time deployment, yet remains challenging. Trajectory-style consistency distillation often becomes conservative under complex video dynamics, yielding an over-smoothed appearance and weak motion. Distribution matching distillation (DMD) can recover sharp, mode-seeking samples, but its local training signals do not explicitly regularize how denoising updates compose across timesteps, making composed rollouts prone to drift. To overcome this challenge, we propose Self-Consistent Distribution Matching Distillation (SC-DMD), which explicitly regularizes the endpoint-consistent composition of consecutive denoising updates. For real-time autoregressive video generation, we further treat the KV cache as a quality parameterized condition and propose Cache-Distribution-Aware training. This training scheme applies SC-DMD over multi-step rollouts and introduces a cache-conditioned feature alignment objective that steers low-quality outputs toward high-quality references. Across extensive experiments on both non-autoregressive backbones (e.g., Wan~2.1) and autoregressive real-time paradigms (e.g., Self Forcing), our method, dubbed \textbf{Salt}, consistently improves low-NFE video generation quality while remaining compatible with diverse KV-cache memory mechanisms. Source code will be released at \href{https://github.com/XingtongGe/Salt}{https://github.com/XingtongGe/Salt}.
Engineering Breakdown
Plain English
This paper tackles the problem of running video generation models with extremely tight computational budgets—specifically 2-4 neural function evaluations (NFEs)—while maintaining visual quality for real-time deployment. The authors identify that existing distillation approaches either produce over-smoothed, low-motion videos (trajectory-style methods) or suffer from accumulated drift across timesteps (distribution matching approaches). They propose Self-Consistent Distribution Matching Distillation (SC-DMD), which explicitly regularizes how consecutive denoising updates compose together, preventing error accumulation while preserving sharp details and realistic motion. The method treats autoregressive video generation as a special case and extends the framework to handle temporal consistency across frames.
Core Technical Contribution
The core innovation is introducing explicit endpoint-consistency regularization into distribution matching distillation—forcing the composition of consecutive denoising updates to maintain alignment rather than allowing them to drift independently. Unlike prior trajectory-style distillation that enforces intermediate step consistency (creating conservative outputs), SC-DMD regularizes the final endpoint reached by chains of denoising operations, allowing mode-seeking behavior while preventing accumulated errors. The key insight is that local training signals in standard DMD don't account for how errors compound across multiple timesteps in rollout generation, whereas SC-DMD directly optimizes for this composition property. This is particularly important for autoregressive video generation where frames are generated sequentially and each frame's quality affects downstream frames.
How It Works
The method operates on a pretrained video generation diffusion model that the authors distill down to 2-4 NFEs. During training, SC-DMD samples pairs of consecutive denoising trajectories and explicitly constrains that following path A then path B produces the same endpoint as following an alternative path, creating a consistency constraint. The distillation process uses a teacher model (full-step diffusion) to guide a student model (low-NFE version) by matching distributions in latent space while enforcing that composed updates don't diverge. For autoregressive video generation, the framework extends this by treating each frame's denoising trajectory as dependent on previously generated frames, and the consistency regularization ensures that frame-to-frame transitions remain stable. The student model learns to make larger, higher-quality denoising steps per NFE compared to naive step reduction, effectively learning which intermediate steps can be skipped without accumulating error.
Production Impact
This directly enables real-time video generation on edge devices and in streaming applications where 2-4 NFEs translates to generation times of 100-500ms on commodity hardware (vs. 5-15 seconds for full models). Engineers deploying video AI systems would replace their standard diffusion pipelines with SC-DMD-distilled variants, reducing memory footprint by 3-4x and inference latency by similar factors while maintaining perceptual quality. The approach integrates cleanly into existing diffusion-based video pipelines—it's a distillation method, not an architectural change, so existing model infrastructure can be reused. Trade-offs include needing a large video dataset and computational budget for the distillation process itself (training can take days on multiple GPUs), and the distilled models are specific to their target NFE count, so deploying different latency targets requires separate distilled models. The consistency regularization adds computational overhead during training but has zero inference cost.
Limitations and When Not to Use This
The paper assumes access to a high-quality teacher model and sufficient compute for distillation, which may not hold for organizations without substantial ML infrastructure. SC-DMD's effectiveness likely degrades on video content significantly different from training distribution (domain shift), and the paper doesn't thoroughly explore failure modes on long-form videos or complex camera motion beyond what's in the training set. The endpoint-consistency constraint is a discrete-step optimization and may not capture continuous temporal consistency properties needed for some applications (e.g., physics-based video editing or motion control). Additionally, the paper appears incomplete in the abstract (cutting off mid-sentence about autoregressive generation), suggesting key methodological details on how autoregressive extension works may not be fully specified, which would be critical for practitioners implementing this.
Research Context
This work extends distribution matching distillation (DMD), which had been proposed to improve mode coverage in diffusion model distillation but suffered from composition drift. SC-DMD builds on a growing body of research in efficient diffusion models, competing with orthogonal approaches like quantization, pruning, and architecture reduction but focusing specifically on distillation quality. The paper contributes to the video generation track of research, building on foundations laid by latent diffusion models for video and recent work on autoregressive frame-by-frame generation. It represents a shift toward production-oriented diffusion research that explicitly addresses the 100-1000ms latency requirement for interactive applications, opening new research directions around compositional consistency constraints in other sequential generation tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::