Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
| Authors | Diyuan Wu et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2603.05691 |
| Download | |
| Categories | cs.LG, stat.ML |
Abstract
It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained from a weak teacher, and yet the strong student outperforms the weak teacher. In this paper, we show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error. Specifically, we consider students and teachers trained via random feature ridge regression (RFRR). Our main technical contribution is to derive a deterministic equivalent for the excess test error of the student trained on labels obtained via the teacher. Via this deterministic equivalent, we then identify regimes in which the scaling law of the student improves upon that of the teacher, unveiling that the improvement can be achieved both in bias-dominated and variance-dominated settings. Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher -- in fact, when the test error of the teacher does not even decay with the sample size.
Engineering Breakdown
Plain English
This paper analyzes weak-to-strong generalization in machine learning—where a strong student model trained on imperfect labels from a weak teacher can outperform the teacher. The authors focus on random feature ridge regression (RFRR) models and derive a deterministic equivalent formula that predicts the excess test error when students learn from teacher-generated labels. Their main finding is that this weak-to-strong setup can produce substantial improvements that actually change the scaling law governing how test error decreases with model capacity, not just shift performance by a constant amount.
Core Technical Contribution
The core technical contribution is deriving a closed-form deterministic equivalent (a rigorous mathematical characterization) for the excess test error of a student model trained on teacher-generated labels in the RFRR setting. This moves beyond empirical observation of weak-to-strong generalization to provide precise theoretical predictions of when and how much a student can exceed teacher performance. The deterministic equivalent captures how label noise from the imperfect teacher propagates through the student's training, enabling prediction of scaling law exponents rather than just point estimates. This is novel because prior work treated weak-to-strong generalization empirically; this paper provides the mathematical machinery to predict improvement magnitude analytically.
How It Works
The mechanism operates in two stages: first, a weak teacher model is trained via random feature ridge regression on original data, producing imperfect predictions; second, a strong student model is also trained via RFRR, but instead of learning from true labels, it learns from the teacher's soft or hard predictions. The key technical move is deriving a deterministic equivalent—an exact (in the large-data limit) formula that describes the student's excess risk as a function of the teacher's error rate, regularization parameters, and data dimensionality. This formula captures how the teacher's label noise affects the student's learning curve and reveals that improvements manifest in the scaling law exponent itself (e.g., error might decay as n^-α instead of n^-β), not merely as a constant factor. The deterministic equivalent likely leverages random matrix theory and the implicit bias properties of ridge regression to sidestep simulation and provide closed-form predictions.
Production Impact
For production systems using weak labeling (crowdsourced labels, model-generated pseudo-labels, weak supervision signals), this provides a theoretical foundation to predict whether a stronger downstream model will actually outperform its labeler and by how much, eliminating trial-and-error. Instead of running expensive re-training experiments to validate if a weak-to-strong pipeline is worth the computational cost, engineers can use the deterministic equivalent to estimate scaling behavior upfront and decide whether to proceed. This is particularly valuable in continual learning or bootstrapping scenarios where you iteratively improve weak signals—you can forecast whether successive model generations will improve faster than linear scaling would suggest. The production trade-off is that these predictions are derived for RFRR specifically, so you must either use RFRR (limiting model flexibility) or empirically validate that the theory holds for your architecture (neural networks, transformers, etc.). Additionally, the theory assumes specific data regimes (large n, known noise structure) that may not match real-world label noise distributions.
Limitations and When Not to Use This
This analysis is restricted to random feature ridge regression—a linear model with fixed random embeddings—which is far simpler than the deep neural networks and transformers used in production systems, and the theory may not transfer to those settings. The deterministic equivalent assumes the large-sample regime (n→∞) and requires knowledge or accurate estimation of the teacher's error rate and noise characteristics, which is difficult when label corruption is heterogeneous or unknown. The paper does not address distribution shift between teacher and student data, online learning scenarios where label distribution changes over time, or scenarios where the student and teacher use different architectures or feature spaces. Follow-up work is needed to extend the theory to non-linear models, characterize robustness to model misspecification, and validate empirically that the deterministic equivalent predictions hold for realistic label noise in large-scale datasets.
Research Context
This paper advances the theoretical understanding of weak-to-strong generalization, a phenomenon that has gained prominence due to its relevance to self-training, pseudo-labeling, and scaling laws in modern deep learning. It builds on foundational random matrix theory results and deterministic equivalent techniques from high-dimensional statistics, applying them to a structured two-stage learning setting. The work connects to recent research on scaling laws and implicit bias in overparameterized models, extending those insights to the transfer setting where one model's output becomes another's input. This opens a research direction toward similar theoretical characterizations for nonlinear models and realistic neural architectures, potentially providing principled guidelines for when and how much to trust weak supervision signals in scaling scenarios.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::