Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
| Authors | Jie Huang et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2604.09412 |
| Download | |
| Categories | stat.ML, cs.LG |
Abstract
We study the population loss landscape of two-layer ReLU networks of the form \sum_{k=1}^K \mathrm{ReLU}(w_k^\top x) in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical structure of minima: they are typically isolated in the well-specified regime, but become connected by flat directions as network width increases. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.
Engineering Breakdown
Plain English
This paper analyzes the loss landscape of two-layer ReLU neural networks trained on Gaussian data in a teacher-student setup where the problem is solvable (realisable setting). The authors prove that local minima can be exactly characterized using low-dimensional summary statistics rather than the full high-dimensional weight space, and show these minima correspond to fixed points of one-pass stochastic gradient descent. A key finding is that the landscape structure changes dramatically with network width: in standard networks, minima are isolated and hard to reach, but as you add more neurons (overparameterisation), flat directions connect the minima together, making global optima much easier for SGD to find.
Core Technical Contribution
The core novelty is providing an exact low-dimensional representation of the loss landscape for two-layer ReLU networks using summary statistics, rather than analyzing weights directly. The authors establish a rigorous connection between local minima in weight space and attractive fixed points of SGD dynamics in this reduced summary statistics space. This reveals a previously uncharacterized hierarchical structure where network width fundamentally reorganizes the landscape topology—moving from isolated local minima to connected manifolds of solutions. This is technically distinct from prior work because it provides both analytical characterization (exact landscape equations) and dynamic interpretation (SGD fixed points) in a unified framework.
How It Works
The method analyzes two-layer networks of the form Σ ReLU(w_k^T x) where w_k are weight vectors and x are Gaussian input features. Instead of working with K×d dimensional weight matrices directly, the authors define summary statistics that capture the essential structure (likely correlations between weight vectors and projections onto the data distribution). They then prove that any local minimum admits an exact representation in this lower-dimensional summary space, reducing the effective dimensionality from exponential to polynomial. For the dynamic component, they track how SGD updates evolve in summary statistics space and show that local minima become fixed points of this reduced dynamics. Finally, they analyze how the connectivity structure changes: in the well-specified regime they prove minima are generically isolated, but as width K increases, flat directions emerge connecting different minima into a manifold.
Production Impact
For engineers, this provides theoretical justification for several practical observations: (1) larger networks (overparameterised) are easier to train because the loss landscape becomes more benign with connected minima, explaining why you often see better convergence with wider networks; (2) you can potentially monitor summary statistics during training instead of tracking all weights, reducing memory overhead and diagnostic complexity for large models; (3) the guaranteed connection to SGD fixed points means you can predict whether your optimization will converge based on landscape geometry before training. However, this applies specifically to two-layer ReLU networks on Gaussian data—extending to deeper networks, other activation functions, or realistic data distributions requires substantial additional work. The practical gain is limited unless you're specifically working on shallow architectures or using this for theoretical analysis of convergence guarantees.
Limitations and When Not to Use This
This analysis is restricted to two-layer ReLU networks, which are rarely used in production; extending to deeper networks (the realistic case) remains open. The Gaussian covariate assumption is strong and doesn't reflect real data distributions—images, text, and other domains have complex structure that would invalidate the landscape characterization. The realisable setting (where a teacher network generates labels) is unrealistic for most applications where you have model misspecification and noise. Additionally, while the paper proves local minima have this structure, it doesn't provide constructive algorithms to find these minima efficiently—the connection to SGD is descriptive rather than prescriptive, so practitioners can't immediately use this to improve optimization algorithms. The summary statistics representation, while theoretically elegant, may require custom derivation for each architecture, limiting generalizability.
Research Context
This work builds on a growing body of theoretical ML research analyzing neural network loss landscapes (following papers on mode connectivity and lottery ticket hypotheses). It extends prior characterizations of overparameterised networks by providing exact landscape equations rather than asymptotic or approximate results. The paper connects to the broader agenda of understanding implicit bias—why SGD generalizes well and finds good solutions—by showing how network width reorganizes the loss geometry. This opens research directions toward: (1) extending the summary statistics approach to deeper networks and non-Gaussian data; (2) developing practical algorithms that leverage the landscape structure for faster convergence; (3) understanding how the landscape changes under different loss functions beyond MSE.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::