A theory of learning data statistics in diffusion models, from easy to hard
| Authors | Lorenzo Bardone et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2603.12901 |
| Download | |
| Categories | stat.ML, cs.IT, cs.LG |
Abstract
While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on natural images exhibit a distributional simplicity bias, learning simple, pair-wise input statistics before specializing to higher-order correlations. We reproduce this behaviour in simple denoisers trained on a minimal data model, the mixed cumulant model, where we precisely control both pair-wise and higher-order correlations of the inputs. We identify a scalar invariant of the model that governs the sample complexity of learning pair-wise and higher-order correlations that we call the diffusion information exponent, in analogy to related invariants in different learning paradigms. Using this invariant, we prove that the denoiser learns simple, pair-wise statistics of the inputs at linear sample complexity, while more complex higher-order statistics, such as the fourth cumulant, require at least cubic sample complexity. We also prove that the sample complexity of learning the fourth cumulant is linear if pair-wise and higher-order statistics share a correlated latent structure. Our work describes a key mechanism for how diffusion models can learn distributions of increasing complexity.
Engineering Breakdown
Plain English
This paper investigates why diffusion models learn data statistics in a particular order—simple pairwise correlations first, then progressively more complex higher-order correlations. The authors demonstrate this empirically on natural images and validate their theory using a controlled synthetic dataset called the mixed cumulant model, where they can precisely tune the complexity of input statistics. They discover a scalar metric called the diffusion information exponent that predicts how many samples are needed to learn each level of statistical complexity. This is the first rigorous characterization of the learning dynamics in diffusion models, moving beyond treating them as black boxes.
Core Technical Contribution
The main novelty is identifying and formalizing the distributional simplicity bias in diffusion models—the phenomenon that these models learn statistics in order of increasing complexity rather than all at once. The authors introduce the diffusion information exponent, a scalar invariant derived from the data distribution and model architecture that quantitatively governs sample complexity for learning different statistical levels. This provides an analytical framework analogous to existing invariants in other learning theory contexts, but specific to the denoising-based training used in diffusion models. Prior work treated diffusion model learning as a black box; this paper opens it with a principled theory that connects data properties to learning speed.
How It Works
The approach has two main components working in tandem. First, empirical validation: the authors train standard diffusion models on natural image datasets and observe via intermediate model checkpoints that early training focuses on low-order correlations (like color statistics), while later training captures fine textures and shapes (high-order correlations). Second, theoretical validation using the mixed cumulant model—a synthetic data distribution where they explicitly control the strength of pairwise versus higher-order cumulants independently. By training simple denoising networks on this synthetic data and measuring convergence, they identify which properties of the input distribution determine the learning order. The diffusion information exponent emerges as a scalar function of these properties that predicts sample complexity empirically, allowing engineers to estimate how much data and compute is needed for a given statistical target without running full training.
Production Impact
This work directly impacts how organizations budget compute and data for diffusion model training. If you understand the diffusion information exponent for your target distribution, you can estimate training curves and stop early if only coarse statistics matter for your downstream task. For example, if you're generating low-resolution images or only need models to capture color and basic shapes, you can measure the exponent and determine you need 10x fewer iterations than a model targeting fine texture synthesis. It also informs data prioritization: you can invest in collecting simple, high-quality examples early, then focus on rare edge cases and complex patterns later. The main trade-off is that computing the exponent requires some analysis of your specific data distribution, and the theory applies most cleanly to synthetic or well-understood domains; highly diverse real-world data may have exponents that are hard to estimate a priori.
Limitations and When Not to Use This
The paper's scope is constrained by its reliance on relatively simple synthetic models (mixed cumulants) that may not capture the full complexity of real image distributions or other modalities like text or audio. The diffusion information exponent is derived for specific architectural choices (standard Gaussian diffusion schedules and simple denoising networks), and it's unclear how robust this ordering is to recent variants like flow matching, variance-weighted objectives, or discrete diffusion. The theory also assumes stationarity and homogeneity in the data statistics—real-world distributions often have highly structured, non-stationary correlations that may violate the model's assumptions. Finally, the paper does not address how to leverage this theory to improve model performance or training efficiency algorithmically; it is primarily descriptive rather than prescriptive.
Research Context
This work builds on decades of learning theory (particularly classical results on learning depth in neural networks and the role of implicit bias) while bringing those insights to the modern era of generative models. It extends the spirit of theoretical analyses in contrastive learning and energy-based models that also identified simplicity bias, but adapts those tools to the denoising objective unique to diffusion models. The paper fits into a growing body of work (e.g., recent analyses of neural scaling laws and grokking phenomena) that aims to characterize which statistics models learn when, rather than just whether they converge. This opens the door for future work on designing curricula, importance sampling, or adaptive objectives that align with the natural learning order, potentially accelerating diffusion model training significantly.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::