Large deviation principles for convolutional Bayesian neural networks
| Authors | Federico Bassetti et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2603.06023 |
| Download | |
| Categories | stat.ML |
Abstract
While suitably scaled CNNs with Gaussian initialization are known to converge to Gaussian processes as the number of channels diverges, little is known beyond this Gaussian limit. We establish a large deviation principle (LDP) for convolutional neural networks in the infinite-channel regime. We consider a broad class of multidimensional CNN architectures characterized by general receptive fields encoded through a patch-extractor function satisfying mild structural assumptions. Our main result establishes a large deviation principle (LDP) for the sequence of conditional covariance matrices under Gaussian prior distribution on the weights. We further derive an LDP for the posterior distribution obtained by conditioning on a finite number of observations. In addition, we provide a streamlined proof of the concentration of the conditional covariances and of the Gaussian equivalence of the network. To the best of our knowledge, this is the first large deviation principle established for convolutional neural networks.
Engineering Breakdown
Plain English
This paper establishes large deviation principles (LDP) for convolutional neural networks operating in the infinite-channel limit, extending beyond the well-known Gaussian process convergence result. The authors prove that CNNs with Gaussian weight initialization satisfy an LDP for their conditional covariance matrices and posterior distributions when conditioned on finite observations. The work applies to a broad class of multidimensional CNN architectures with general receptive fields, providing theoretical guarantees about how the network's uncertainty behaves as channel width grows. This fills a significant gap: while we know CNNs converge to GPs in the limit, this paper characterizes rare event probabilities and posterior concentration rates—critical for understanding Bayesian neural network behavior at scale.
Core Technical Contribution
The main novelty is proving a large deviation principle for the sequence of conditional covariance matrices of infinite-width CNNs under a Gaussian prior, which quantifies the exponential rate at which the posterior concentrates around the true parameter. Prior work established that CNNs converge to Gaussian processes, but said nothing about the probability of deviations from this limiting behavior or how quickly the posterior learns. The authors achieve this through a general framework that handles multidimensional architectures with arbitrary receptive field structures encoded via a patch-extractor function, rather than restricting to standard convolutional patterns. They further extend the LDP to the posterior distribution itself, enabling concentration rate analysis when the network observes finite training data—a result not previously available for this class of models.
How It Works
The paper begins by parameterizing a CNN through a patch-extractor function that generalizes how convolutional operations extract spatial patches, allowing analysis of diverse receptive field geometries beyond standard convolutions. For an infinite-channel CNN with Gaussian weight initialization, the authors model the conditional covariance (the epistemic uncertainty given the architecture) as a stochastic process indexed by input locations. They establish that this sequence of covariance matrices satisfies a large deviation principle: roughly, the probability of observing deviations from the typical covariance behavior decays exponentially at a rate governed by a rate function. In the posterior setting, conditioning on n observations, the authors show the posterior concentrates with exponential speed, and the rate function sharpens—the network becomes more confident in its learned representations. The core technical machinery involves contraction principles, weak convergence arguments, and careful handling of the infinite-dimensional function space topology.
Production Impact
For engineers building Bayesian deep learning systems, this work provides theoretical justification for confidence in infinite-width CNN posteriors: you can now rigorously quantify how fast the network learns and how reliable its uncertainty estimates are. In practice, this means you can set principled stopping criteria for approximate posterior inference (e.g., variational or MCMC methods)—once n observations exceed certain thresholds tied to the rate function, you have exponential concentration guarantees. The patch-extractor generalization is valuable for practitioners working with non-standard receptive fields (dilated convolutions, grouped convolutions, or custom spatial operations), as the theory directly applies. However, the infinite-channel limit is asymptotic; you'd still need to empirically validate that your finite-width networks (typically 32–512 channels in production) exhibit the predicted concentration behavior. The computational cost is purely theoretical—this is analysis, not a new training algorithm—so integration requires only updating uncertainty quantification validation protocols.
Limitations and When Not to Use This
The primary limitation is that all results assume Gaussian weight initialization under a Gaussian prior, which is restrictive compared to modern practice using ReLU networks with non-Gaussian initialization schemes (He initialization, etc.). The infinite-channel limit is asymptotic; the paper provides no explicit rates of convergence for finite widths, so practitioners cannot directly predict how many channels are needed to see the theoretical guarantees in action. The theory assumes a fully observed, finite training set; it does not address online learning, distribution shift, or highly imbalanced data regimes common in production. Additionally, the patch-extractor framework, while general, requires the architectural structure to be Markovian in spatial structure—truly arbitrary attention-based interactions or fully-connected components fall outside the scope. The paper does not provide algorithms for computing the rate function explicitly for arbitrary architectures, limiting applicability to practitioners without deep mathematical expertise.
Research Context
This work builds directly on the infinite-width limit theory initiated by Neal (1996) and extended to CNNs by Novak et al. and Jacot et al., who showed CNNs converge to Gaussian processes as channel width diverges. The large deviation principle framework is borrowed from classical probability theory but applied here in a novel way to neural network posterior analysis. The paper advances the frontier of neural network theory beyond asymptotic behavior (Gaussian process limits) into rare-event asymptotics, analogous to how large deviation theory in statistical mechanics characterizes phase transitions. This opens new research directions: understanding LDPs for other architectures (RNNs, Transformers), non-Gaussian initializations, and the computational aspects of verifying these principles in practice. The rate function derived could inform future work on neural network scaling laws and optimal architecture design for learning efficiency.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::