When Your Model Stops Working: Anytime-Valid Calibration Monitoring
| Authors | Tristan Farran |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2603.13156 |
| Download | |
| Categories | stat.ME, stat.ML |
Abstract
Practitioners monitoring deployed probabilistic models face a fundamental trap: any fixed-sample test applied repeatedly over an unbounded stream will eventually raise a false alarm, even when the model remains perfectly stable. Existing methods typically lack formal error guarantees, conflate alarm time with changepoint location, and monitor indirect signals that do not fully characterize calibration. We present PITMonitor, an anytime-valid calibration-specific monitor that detects distributional shifts in probability integral transforms via a mixture e-process, providing Type I error control over an unbounded monitoring horizon as well as Bayesian changepoint estimation. On river's FriedmanDrift benchmark, PITMonitor achieves detection rates competitive with the strongest baselines across all three scenarios, although detection delay is substantially longer under local drift.
Engineering Breakdown
Plain English
This paper addresses a critical problem in production ML: when you repeatedly test a deployed probabilistic model for calibration drift using standard statistical tests, you will eventually get false alarms even when the model is working perfectly. The authors present PITMonitor, a monitoring system that detects real distributional shifts in a model's probability predictions while maintaining formal guarantees that false alarm rates stay under control indefinitely—no matter how long you monitor. The approach uses probability integral transforms (a mathematically sound way to check if predictions match reality) combined with mixture e-processes (a sequential testing technique). On the FriedmanDrift benchmark, PITMonitor achieves competitive detection rates while providing the theoretical guarantees that existing methods lack.
Core Technical Contribution
The core novelty is combining two ideas that haven't been properly integrated before: (1) using probability integral transforms (PIT) as the direct signal for monitoring calibration rather than indirect proxies, and (2) applying anytime-valid e-process-based hypothesis testing to maintain Type I error control over unbounded monitoring horizons. E-processes are a recent theoretical tool that solve the multiple-testing problem—they let you keep testing indefinitely without accumulating false positives, unlike classical p-values which degrade under repeated testing. The paper also adds Bayesian changepoint estimation, so when the monitor triggers, you get a principled estimate of when the shift actually occurred, not just when the alarm fired. This combination is novel because prior drift detection methods either lacked formal error guarantees or conflated detection time with true change location.
How It Works
The system operates on a continuous stream of predictions from a deployed model. For each prediction, the model outputs a probability distribution, and the observed true label arrives later; the algorithm computes the probability integral transform (PIT) for each prediction-label pair—this is a value between 0 and 1 that should be uniformly distributed if the model is perfectly calibrated. These PIT values are fed into a mixture e-process, which is a sequential statistical test designed to detect when the distribution of PIT values drifts away from uniform. The e-process maintains a likelihood ratio-like value that grows when evidence of shift accumulates, but crucially, it uses a mixture over possible changepoint locations so the final detection naturally includes Bayesian uncertainty about when the shift occurred. When the e-process exceeds a threshold, an alarm fires and you get both a detection flag and a posterior distribution over possible changepoint times.
Production Impact
For engineers monitoring live models, this solves a painful real problem: today's monitoring dashboards either use naive repeated testing (generating phantom alerts that erode trust) or use heuristic checks without error guarantees (leaving you uncertain if an alert is real). PITMonitor lets you deploy a monitor and trust that legitimate calibration failures will be caught while false positives remain provably rare over months or years of continuous operation. The practical workflow changes: instead of setting ad-hoc alert thresholds on model confidence metrics, you set a single false-alarm rate budget (e.g., one false alarm per year), and the monitor automatically adapts to the data stream. Trade-offs include modest computational overhead (computing PIT values and running the e-process for each prediction is cheap but non-zero), a requirement that you have ground-truth labels available to compute PIT, and a delay before changes are detected (the e-process needs to accumulate sufficient evidence). Integration is straightforward if your pipeline already computes prediction probabilities and logs labels.
Limitations and When Not to Use This
The method requires true labels to be available for computing PIT values, which isn't always realistic in production—if labels arrive very slowly or sparsely, detection latency increases significantly. The approach is specifically designed for calibration shifts, not for detecting other important distribution shifts (like input distribution changes or accuracy collapse), so you may still need complementary monitors. The paper assumes the underlying model produces well-formed probability distributions; if the model's uncertainty estimates are fundamentally broken, PIT-based monitoring may not help. The theoretical guarantees are asymptotic and hold over an unbounded horizon, but finite-sample behavior and practical threshold-setting details appear incomplete in the abstract. Additionally, while Bayesian changepoint estimation is valuable, the accuracy of pinpointing changepoints depends on having enough post-shift data, so very rapid transient shifts might be flagged late.
Research Context
This work builds on recent advances in anytime-valid testing and e-processes (tools that solve the multiple-testing problem in sequential settings), extending them to the specific domain of model calibration monitoring. It treats the benchmark set from river (a streaming ML evaluation framework) as a standard, which is appropriate for drift detection research but may underrepresent real-world production deployment patterns. The paper advances the broader research direction of statistically rigorous monitoring for deployed ML systems—moving away from heuristic dashboards toward principled inference. It opens future work on extending anytime-valid methods to other model properties (like sharpness, coverage, or fairness) and on handling sparse or delayed label arrival in production settings.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::