Statistical Inference for Score Decompositions
| Authors | Timo Dimitriadis & Marius Puke |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2603.04275 |
| Download | |
| Categories | stat.ME, stat.ML |
Abstract
We introduce inference methods for score decompositions, which partition scoring functions for predictive assessment into three interpretable components: miscalibration, discrimination, and uncertainty. Our estimation and inference relies on a linear recalibration of the forecasts, which is applicable to general multi-step ahead point forecasts such as means and quantiles due to its validity for both smooth and non-smooth scoring functions. This approach ensures desirable finite-sample properties, enables asymptotic inference, and establishes a direct connection to the classical Mincer-Zarnowitz regression. The resulting inference framework facilitates tests for equal forecast calibration or discrimination, which yield three key advantages. They enhance the information content of predictive ability tests by decomposing scores, deliver higher statistical power in certain scenarios, and formally connect scoring-function-based evaluation to traditional calibration tests, such as financial backtests. Applications demonstrate the method's utility. We find that for survey inflation forecasts, discrimination abilities can differ significantly even when overall predictive ability does not. In an application to financial risk models, our tests provide deeper insights into the calibration and information content of volatility and Value-at-Risk forecasts. By disentangling forecast accuracy from backtest performance, the method exposes critical shortcomings in current banking regulation.
Engineering Breakdown
Plain English
This paper introduces a statistical framework for decomposing scoring functions used in predictive modeling into three interpretable components: miscalibration (how well predictions match actual outcomes), discrimination (ability to distinguish between different outcomes), and uncertainty (inherent randomness). The authors propose using linear recalibration of forecasts as the foundation for inference, which works for both smooth functions (like mean squared error) and non-smooth functions (like quantile loss), enabling rigorous hypothesis testing on forecast quality. Their method connects classical econometric techniques (Mincer-Zarnowitz regression) with modern scoring decompositions, providing finite-sample validity and asymptotic inference properties that practitioners can rely on. The practical benefit is that forecasters can now statistically test whether one model is better calibrated or more discriminative than another, moving beyond point estimates to formal statistical comparisons.
Core Technical Contribution
The core innovation is a theoretically grounded decomposition framework that partitions any scoring function into orthogonal components tied to specific forecast quality dimensions, coupled with a linear recalibration procedure that yields valid inference for both smooth and non-smooth scoring rules. Unlike prior decomposition approaches that may lack statistical inference capabilities or be limited to specific loss functions, this method establishes finite-sample validity and asymptotic normality, allowing confidence intervals and hypothesis tests on miscalibration and discrimination differences. The key technical contribution is proving that linear recalibration preserves the decomposition structure and validity across diverse scoring functions commonly used in practice (point forecasts, quantiles, probabilistic forecasts). This bridges a gap between classical econometric regression diagnostics and modern machine learning evaluation frameworks by showing how to rigorously test forecast improvements.
How It Works
The method begins with a scoring function S(y, ŷ) that measures forecast quality (e.g., MSE, quantile loss, log loss), and the authors decompose it into three orthogonal components corresponding to miscalibration, discrimination, and uncertainty. The inference procedure involves fitting a linear recalibration model on the forecast outputs, which maps the original forecasts to a recalibrated form that optimizes the expected scoring function; this recalibration is equivalent to the Mincer-Zarnowitz regression in the time-series context. For each scoring function, the authors derive the gradient of the score with respect to forecasts, which characterizes how predictions influence the three decomposition components; linear recalibration exploits this structure to create estimates with known asymptotic distributions. The output is a set of estimated decomposition coefficients with corresponding standard errors and confidence intervals, enabling hypothesis tests like H₀: discrimination is equal between two forecasts, or H₀: model A is not less miscalibrated than model B. The entire framework works for multi-step ahead forecasts (not just one-step) and accommodates any forecast type (conditional means, quantiles, probability distributions) because the mathematics applies to any scoring function with bounded derivatives.
Production Impact
In production forecasting systems, this enables data scientists to move beyond comparing raw forecast metrics (e.g., 'Model A has MAE=2.3, Model B has MAE=2.5') to rigorous statistical testing of whether observed differences are significant or just noise. Teams deploying quantile or probabilistic forecasts gain a principled way to assess which model components (calibration, discrimination) are driving performance differences, allowing targeted improvements—for example, if discrimination is weak but calibration is good, focus on feature engineering rather than recalibration techniques. The linear recalibration step is computationally cheap (closed-form solution, no iterative optimization) and can be applied post-hoc to existing forecasts without retraining, making it easy to integrate into existing ML pipelines for forecast comparison and validation. Trade-offs include modest additional computational cost during evaluation (computing recalibration coefficients and decomposition statistics), the requirement for held-out test data to compute valid confidence intervals, and the assumption that the linear recalibration model correctly captures forecast adjustments (which may not hold if the true relationship is highly nonlinear, though the authors' theory suggests it's robust). The practical value is highest in regulated domains (finance, weather, electricity) where forecast quality improvements must be statistically defensible.
Limitations and When Not to Use This
The paper's analysis assumes that the linear recalibration model is sufficient to capture systematic forecast bias, which may fail in regimes with complex nonlinear relationships between forecast features and errors, or when the data distribution shifts significantly post-deployment. The decomposition is based on expected values of scoring functions, so it may not directly address tail-risk scenarios or performance in extreme quantiles, which matter for applications like rare-event prediction or anomaly detection. The finite-sample validity properties rely on standard regularity conditions (bounded moments, ergodic dependence structures) that may not hold for all real-world data, particularly in highly correlated time series or data with structural breaks. The paper also does not extensively address how to choose between competing decompositions or how the framework adapts when the scoring function itself is misspecified for the application (e.g., using MSE when quantile loss better matches business objectives).
Research Context
This work builds on a long tradition of forecast evaluation going back to the Mincer-Zarnowitz regression (1969), which tested forecast rationality in economics, and extends it through modern statistical frameworks for evaluating probabilistic and point forecasts developed by Gneiting, Raftery, and others in the 2000s-2010s. The decomposition idea connects to broader literature on forecast skill measures and calibration assessment, particularly work by Murphy (1971) on decomposing Brier scores into reliability and resolution, and extends those principles to arbitrary scoring rules. By providing asymptotic inference theory for score decompositions, the paper addresses a practical gap: practitioners have had tools to decompose forecasts post-hoc, but lacked formal statistical tests for comparing components across models. The contribution opens directions for future work on: (1) nonlinear recalibration schemes with similar inference properties, (2) extensions to multivariate forecasts and forecast combinations, and (3) robustness to distribution shifts or covariate shifts in real deployment scenarios.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::