From Shallow Bayesian Neural Networks to Gaussian Processes: General Convergence, Identifiability and Scalable Inference
| Authors | Gracielle Antunes de Araújo & Flávio B. Gonçalves |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2602.22492 |
| Download | |
| Categories | stat.ML, cs.AI, cs.LG |
Abstract
In this work, we study scaling limits of shallow Bayesian neural networks (BNNs) via their connection to Gaussian processes (GPs), with an emphasis on statistical modeling, identifiability, and scalable inference. We first establish a general convergence result from BNNs to GPs by relaxing assumptions used in prior formulations, and we compare alternative parameterizations of the limiting GP model. Building on this theory, we propose a new covariance function defined as a convex mixture of components induced by four widely used activation functions, and we characterize key properties including positive definiteness and both strict and practical identifiability under different input designs. For computation, we develop a scalable maximum a posterior (MAP) training and prediction procedure using a Nyström approximation, and we show how the Nyström rank and anchor selection control the cost-accuracy trade-off. Experiments on controlled simulations and real-world tabular datasets demonstrate stable hyperparameter estimates and competitive predictive performance at realistic computational cost.
Engineering Breakdown
Plain English
This paper establishes a rigorous mathematical bridge between shallow Bayesian neural networks and Gaussian processes by proving convergence under relaxed assumptions compared to prior work. The authors propose a new covariance function that mixes components from four common activation functions (ReLU, sigmoid, tanh, linear) and prove it's a valid positive definite kernel. They develop a scalable maximum a posteriori (MAP) inference procedure that addresses the computational bottleneck of exact GP inference, making the theoretical connection practically useful for production systems that need both Bayesian uncertainty quantification and computational efficiency.
Core Technical Contribution
The paper makes three distinct technical contributions: (1) a generalized convergence proof from BNNs to GPs that removes restrictive assumptions from the literature, enabling broader applicability to different network architectures and initializations; (2) a novel convex mixture kernel combining four activation function-induced kernels with theoretical guarantees on positive definiteness and identifiability; (3) a scalable MAP training algorithm that bypasses the O(n³) cost of exact GP inference while preserving Bayesian posterior approximation quality. The identifiability analysis distinguishes between strict identifiability (parameter recovery under infinite data) and practical identifiability (parameter recovery with finite data and regularization), providing engineers with concrete guidance on when parameters can actually be learned.
How It Works
The method starts by taking a shallow Bayesian neural network with a hidden layer of m neurons, random fixed weights in the first layer, and learned weights in the output layer, plus a Gaussian prior on the weights. As the hidden layer width m scales to infinity with appropriate weight scaling, the network output converges in distribution to a Gaussian process whose covariance kernel is determined by the activation function. The authors then construct a new kernel as K(x,x') = Σᵢ αᵢ Kᵢ(x,x') where each Kᵢ is the limiting kernel from a different activation function and αᵢ ≥ 0 sum to one. For inference, instead of computing the exact GP posterior (which requires inverting an n×n covariance matrix), they use MAP optimization: maximize log p(y|X,θ) + log p(θ) jointly over hyperparameters θ and predictions, using modern optimization like L-BFGS or stochastic gradient descent. Predictions follow standard GP form but with learned hyperparameters, avoiding matrix inversion bottlenecks.
Production Impact
For engineers building systems that need uncertainty quantification without massive computational overhead, this provides a theoretically grounded alternative to standard GPs and BNNs. If you're currently using exact GPs on datasets with thousands of points, you hit the O(n³) memory and O(n³) compute wall; this approach replaces that with MAP optimization that's effectively O(n) to O(n²) depending on optimization algorithm. The identifiability theory tells you whether your data and regularization are sufficient to learn the mixing weights αᵢ reliably—if strict identifiability fails, you know to either increase data or fix some weights, avoiding wasted tuning effort. The multi-kernel mixture approach is particularly valuable for practitioners who don't want to commit to a single activation function or don't know a priori which kernel is best; you get automatic adaptation. Real drawback: MAP gives point estimates of hyperparameters, not a full posterior distribution over them, so you lose some Bayesian uncertainty quantification at the hyperparameter level—acceptable if you're mainly interested in aleatoric uncertainty on predictions.
Limitations and When Not to Use This
The paper assumes a shallow (single hidden layer) network architecture, so it doesn't extend to deep networks where the infinite-width limit behaves differently—practitioners using modern deep architectures won't get the theoretical convergence guarantees. The convergence proof likely requires standard assumptions like sub-Gaussian weight priors and bounded activations, which fail for unbounded activations like ReLU in some regimes, though the paper claims to relax prior assumptions without fully detailing which assumptions remain. Identifiability analysis depends heavily on the input design (are features uniform? clustered? high-dimensional?); the paper characterizes this but doesn't provide algorithms to diagnose identifiability for a given dataset before investing in training. The scalable inference uses MAP optimization which is a point estimate, not full posterior sampling, so posterior uncertainty over learned hyperparameters is unavailable—if you need uncertainty over kernel choices, you'd need additional techniques like Laplace approximation or MCMC.
Research Context
This work builds directly on the neural network GP correspondence literature (Neal 1996, Jacot et al. 2019 on Neural Tangent Kernels) by removing restrictions on parameter initialization and providing more general convergence theorems. The convex mixture kernel extends classical GP kernel design by automating the selection of activation-specific kernels rather than requiring manual specification, complementing kernel learning literature. The identifiability analysis connects to broader statistical theory on when parameters can be recovered from data, particularly relevant as GPs with flexible kernels have become more popular in machine learning (via spectral methods, neural kernel networks). The scalable inference contribution speaks to the growing need for GP methods to handle modern data scales—complementing inducing point approximations (Quiñonero-Candela & Rasmussen) and sparse GPs, this MAP-based approach offers a different speed-accuracy tradeoff that may suit practitioners who prioritize interpretability over asymptotic optimality.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::