The Bernstein-von Mises theorem for Bayesian one-pass online learning
| Authors | Jeyong Lee et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2604.27442 |
| Download | |
| Categories | stat.ML |
Abstract
Bayesian online learning provides a coherent framework for sequential inference. However, its theoretical understanding remains limited, particularly in the one-pass setting. Existing theoretical guarantees typically require the mini-batch sample size to diverge, a condition that fails in the one-pass regime. In this paper, we propose a new Bayesian online learning algorithm tailored to the one-pass setting, which incorporates a warm-start phase to ensure stable sequential updates. For this algorithm, we show that the sequentially updated posterior attains the optimal convergence rate. Building on this, we establish an online analogue of the Bernstein-von Mises theorem, which guarantees valid uncertainty quantification without diverging mini-batch sample sizes. Our analysis is based on a novel theoretical framework that differs fundamentally from existing approaches in the online learning literature. Numerical experiments on generalized linear models show that the proposed method matches the performance of the batch estimator while outperforming existing online procedures.
Engineering Breakdown
Plain English
This paper addresses a fundamental gap in Bayesian online learning theory: how to perform valid statistical inference when processing data in a single pass without requiring batch sizes to grow over time. The authors develop a new algorithm that uses a warm-start initialization phase followed by sequential posterior updates, and prove it converges at optimal rates while maintaining valid uncertainty quantification. The key result is an online version of the Bernstein-von Mises theorem, which guarantees that posterior distributions remain properly calibrated for uncertainty quantification even in the challenging one-pass regime where you see each data point exactly once.
Core Technical Contribution
The primary novelty is proving that Bayesian posterior inference can achieve theoretically optimal convergence rates and valid uncertainty quantification in one-pass online settings without requiring diverging mini-batch sizes—a condition that was previously thought necessary. The authors' key insight is the warm-start phase: initializing the posterior with sufficient preliminary data ensures that subsequent sequential updates remain stable and well-behaved even when batch sizes are fixed and small. They establish the first online analogue of the classical Bernstein-von Mises theorem, which is a foundational result in frequentist asymptotics, extending it to the sequential, non-i.i.d. setting. This theoretical contribution fills a gap between classical batch Bayesian theory and the practical needs of streaming inference systems.
How It Works
The algorithm operates in two phases: a warm-start phase where the posterior is initialized using a fixed amount of data, establishing a stable baseline distribution, and a sequential update phase where subsequent mini-batches refine the posterior without requiring batch sizes to grow. In the sequential phase, each new batch of data is used to update the posterior distribution using standard Bayesian updating rules (typically via variational inference or particle filters), with the critical property that the batch size remains constant. The warm-start phase is crucial because it provides sufficient concentration of the initial posterior so that downstream updates don't accumulate error—without it, fixed batch sizes lead to posterior diffusion where uncertainty grows unboundedly over time. The convergence analysis tracks both the statistical error (how well the posterior concentrates around the true parameter) and the sequential bias (how much posterior updates deviate from the true trajectory), proving both vanish at optimal rates. The Bernstein-von Mises result shows that the sequentially updated posterior is asymptotically normal and centered at the maximum likelihood estimate, enabling practitioners to treat the posterior mean as a point estimator with valid confidence intervals.
Production Impact
For production streaming inference systems, this work justifies using Bayesian methods for online learning with fixed computational budgets per update—you no longer need the batch processing overhead that previous theory required. This enables real-time decision systems (recommendation engines, anomaly detection, adaptive control) to maintain properly calibrated uncertainty estimates without exponential growth in variance, directly improving reliability of confidence-based decisions. The warm-start requirement has practical implications: you need to buffer enough initial data (say 1000-10000 samples) before deploying the sequential learner, but after that you can process data in constant-memory streaming fashion with fixed per-step compute. The theory guarantees you can safely use posterior quantiles for downstream tasks like uncertainty-aware ranking or threshold-based alerts, reducing the need for post-hoc calibration. Trade-offs include the upfront cost of the warm-start phase (adds latency to first predictions) and the assumption that your model class is correctly specified—if your model is misspecified, the theoretical guarantees degrade.
Limitations and When Not to Use This
The paper assumes the statistical model is correctly specified—if the true data distribution doesn't belong to your model class, convergence guarantees may not hold or may require much longer sequences. The warm-start requirement means you cannot deploy true one-pass learning from the very first sample; you must accumulate and buffer initial data, which contradicts the motivation of truly streaming systems with extreme memory constraints. The analysis likely focuses on parametric models or specific model classes, so extension to deep learning or highly nonparametric models remains open and potentially difficult—the dependence on model complexity is not fully characterized. The work appears to be theoretical without extensive empirical validation on real datasets or comparison to practical streaming inference systems (particle filters, variational autoencoders, neural network posteriors), leaving questions about whether the theoretical rates translate to wall-clock speedups in practice.
Research Context
This work extends classical asymptotic statistics (the Bernstein-von Mises theorem, dating to mid-20th century) into the online learning regime, building on recent advances in online Bayesian inference and sequential analysis. It relates to the convergence rate literature in online learning, where matching lower and upper bounds typically require assumptions about batch growth that this work relaxes. The result sits at the intersection of frequentist statistical theory and Bayesian inference, contributing to the deeper understanding of when and why Bayesian posteriors provide valid frequentist inference guarantees. This opens research directions in extending these results to broader model classes (deep generative models, infinite-dimensional function spaces), handling model misspecification, and developing practical streaming inference algorithms that provably maintain these theoretical guarantees under computational constraints.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::