Uncertainty Quantification Via the Posterior Predictive Variance
| Authors | Sanjay Chaudhuri et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2603.19804 |
| Download | |
| Categories | stat.ML |
Abstract
We use the law of total variance to generate multiple expansions for the posterior predictive variance. These expansions are sums of terms involving conditional expectations and conditional variances and provide a quantification of the sources of predictive uncertainty. Since the posterior predictive variance is fixed given the model, it represents a constant quantity that is conserved over these expansions. The terms in the expansions can be assessed in absolute or relative sense to understand the main contributors to the length of prediction intervals. We quantify the term-wise uncertainty across expansions varying in the number of terms and the order of conditionates. In particular, given that a specific term in one expansion is small or zero, we identify the other terms in other expansions that must also be small or zero. We illustrate this approach to predictive model assessment in several well-known models.
Engineering Breakdown
Plain English
This paper develops a mathematical framework for understanding where prediction uncertainty comes from in machine learning models by decomposing the posterior predictive variance into multiple interpretable components using the law of total variance. Rather than treating prediction intervals as a black box, the authors show how to break down the total uncertainty into conditional expectations and variances across different ordering of variables, creating multiple alternative expansions that all conserve the same total variance. This decomposition lets practitioners identify which sources of uncertainty (model parameters, latent variables, measurement noise) dominate in their specific predictions, and compare contributions across different mathematical formulations. The key insight is that while the posterior predictive variance is fixed, understanding how it decomposes across different conditional structures provides actionable information for improving model design and uncertainty quantification.
Core Technical Contribution
The core innovation is a systematic method for generating and comparing multiple decompositions of posterior predictive variance using recursive application of the law of total variance, where each decomposition reveals different interpretations of uncertainty sources. Prior work typically computed posterior predictive variance as a single quantity without structural analysis; this paper proves that the same variance can be expressed as sums of conditional expectations and variances in many different ways, and these alternative decompositions are mathematically conserved (sum to the same total). The authors develop the machinery to assess individual terms within and across decompositions, identifying which terms dominate and which become negligible given specific conditions or model configurations. This provides a principled way to prioritize which sources of uncertainty to reduce and which conditional orderings reveal the most actionable structure.
How It Works
The method starts with the posterior predictive variance under a fitted model, which is a fixed scalar quantity. The authors apply the law of total variance recursively: Var[Y] = E[Var[Y|Z]] + Var[E[Y|Z]], choosing different latent variables Z (model parameters, intermediate representations, etc.) at each step to create alternative decompositions. Each decomposition yields terms that must sum to the original variance, but each term now has an interpretable meaning—for instance, E[Var[Y|θ]] represents irreducible aleatoric uncertainty while Var[E[Y|θ]] represents reducible epistemic uncertainty from parameter uncertainty. The algorithm systematically varies the order and set of conditionates (which variables appear in the conditional) to generate different expansions, then computes the contribution of each term both in absolute variance units and as a relative percentage. Finally, the method identifies when specific terms are small or zero across different expansions, which reveals dependencies and redundancies in how the model sources uncertainty.
Production Impact
For practitioners building uncertainty quantification systems, this provides a diagnostic tool to understand which components of your model are actually driving prediction intervals, enabling targeted improvements rather than black-box hyperparameter tuning. If you discover that E[Var[Y|θ]] dominates, you know aleatoric uncertainty is the bottleneck and collecting more/better data won't help—you need different features or a more expressive model class; conversely, if Var[E[Y|θ]] dominates, ensemble methods or Bayesian approaches to parameter uncertainty become high-ROI. In production, you could compute these decompositions offline on a validation set to categorize your predictions (which are parameter-uncertainty-driven vs. noise-driven) and route them accordingly—e.g., retraining vs. data collection workflows. The method adds negligible computational cost if you're already maintaining posterior samples or approximations, but requires multiple forward passes to estimate conditional expectations and variances, so latency-critical systems would compute decompositions asynchronously. Integration into existing Bayesian inference pipelines (Stan, PyMC, TensorFlow Probability) is straightforward since these libraries already expose conditional distributions.
Limitations and When Not to Use This
The paper assumes you have access to the full posterior distribution or posterior samples, which is not always practical for high-dimensional problems or models trained with standard deep learning optimizers—the method degrades to approximations in these cases but the abstract doesn't discuss approximation error bounds. The decomposition requires choosing which variables to condition on, and the paper doesn't provide principled guidance on which orderings will be most interpretable for a given problem; different orderings can yield wildly different-looking breakdowns of the same variance, creating analysis paralysis. The method is fundamentally post-hoc diagnosis rather than a way to improve uncertainty estimates directly, so it doesn't reduce uncertainty—it only helps you understand where it comes from; if all major sources are unavoidable (e.g., inherent data noise), the insights may not be actionable. Finally, the abstract appears incomplete and doesn't specify computational complexity, convergence guarantees for the term-wise uncertainty quantification, or empirical validation on realistic problems, leaving questions about when this diagnostic actually changes decisions compared to simpler sensitivity analyses.
Research Context
This work extends classical statistical decomposition methods (law of total variance) into the modern Bayesian deep learning context, building on decades of work in posterior inference and uncertainty quantification literature. It relates to prior research on epistemic vs. aleatoric uncertainty decomposition (Kendall & Gal, 2017) but provides a more systematic mathematical framework and allows multiple alternative decompositions rather than a single fixed partition. The paper likely benchmarks against or complements existing uncertainty quantification methods in Bayesian neural networks, ensemble methods, and conformal prediction, though the abstract doesn't specify which datasets or tasks. The direction opens up research into automated selection of 'good' decompositions, interpretability measures for comparing expansions, and connections to causal models where conditioning on different variable orderings reveals causal structure in how uncertainty propagates through a model.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::