Model Agreement via Anchoring
| Authors | Eric Eaton et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.23360 |
| Download | |
| Categories | cs.LG, cs.AI |
Abstract
Numerous lines of aim to control \textit{model disagreement} -- the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions between two models trained on independent samples, without any coordination of the training processes. We would like to be able to drive disagreement to zero with some natural parameter(s) of the training procedure using analyses that can be applied to existing training methodologies. We develop a simple general technique for proving bounds on independent model disagreement based on \textit{anchoring} to the average of two models within the analysis. We then apply this technique to prove disagreement bounds for four commonly used machine learning algorithms: (1) stacked aggregation over an arbitrary model class (where disagreement is driven to 0 with the number of models being stacked) (2) gradient boosting (where disagreement is driven to 0 with the number of iterations ) (3) neural network training with architecture search (where disagreement is driven to 0 with the size of the architecture being optimized over) and (4) regression tree training over all regression trees of fixed depth (where disagreement is driven to 0 with the depth of the tree architecture). For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss -- but then show that all of our results generalize to multi-dimensional regression with any strongly convex loss.
Engineering Breakdown
Plain English
This paper addresses the problem of model disagreement—the divergence in predictions between two independently trained machine learning models—and proposes a technique called anchoring to control and minimize this disagreement. Rather than requiring explicit coordination between training processes, the authors develop a general analytical framework that bounds disagreement by anchoring the analysis to the average of two model predictions. The key insight is that this anchoring approach can be applied to existing training methodologies without modification, making it practical for deployment. The work provides theoretical guarantees on how disagreement scales with properties of the training procedure, enabling engineers to tune hyperparameters to drive disagreement toward zero.
Core Technical Contribution
The core novelty is the anchoring-based analysis technique for proving tight bounds on independent model disagreement in real-valued prediction problems. Unlike prior work that either requires coordinated training or relies on strong distributional assumptions, this approach uses the ensemble average as a reference point within the theoretical analysis, allowing disagreement bounds to be derived for standard training procedures without modification. The technique is general enough to apply across different model classes and training methodologies, fundamentally shifting how we can reason about multi-model robustness. This represents a departure from previous methods by decoupling the analytical framework from the training algorithm itself.
How It Works
The mechanism operates by considering two models trained independently on disjoint samples and measuring their expected squared prediction difference. The anchoring procedure introduces the arithmetic mean of the two model predictions as a pivot point in the analysis—this mean serves as a reference for bounding how far each individual model can deviate. Mathematically, disagreement is bounded by decomposing it relative to this anchor point and leveraging concentration inequalities on the individual model errors. The key insight is that the average prediction is a stable target: if both models are trained similarly, their average has favorable properties that constrain how much they can simultaneously diverge. By analyzing the worst-case behavior through this lens, the authors derive parameter-dependent bounds that scale with training set size, model capacity, and regularization strength, allowing practitioners to set hyperparameters that control disagreement.
Production Impact
In production systems, this enables principled ensemble design without the computational cost of coordinated training or model distillation. Engineers can train multiple models independently using their standard pipeline, then apply this framework to predict and control their disagreement, which directly impacts uncertainty quantification and robustness. For safety-critical applications (autonomous systems, medical diagnosis, financial models), bounding disagreement provides formal guarantees on decision consistency—you can now prove that two models will agree with high probability on the same prediction. The practical trade-off is minimal: you gain theoretical guarantees on ensemble diversity at the cost of understanding your training procedure's properties (sample complexity, regularization effects). Integration is straightforward since it doesn't modify the training algorithm itself; it's purely an analytical tool applied post-hoc to bound disagreement between independently trained models.
Limitations and When Not to Use This
The paper's applicability is constrained to real-valued prediction problems, leaving classification and structured prediction as open questions requiring different analytical techniques. The bounds depend on having accurate characterizations of model capacity and training data properties, which are often unknown or loose in practice—this limits how tight the guarantees are for complex, high-dimensional models like deep networks. The analysis assumes independent training on disjoint samples, which doesn't account for correlated errors that arise from shared inductive biases or architectural similarities between models. Additionally, the framework provides worst-case bounds that may be highly conservative, potentially overestimating disagreement and leading to unnecessarily restrictive hyperparameter choices in practice.
Research Context
This work sits at the intersection of ensemble methods, learning theory, and uncertainty quantification—building on classical results in PAC learning and empirical process theory. It advances the literature on model disagreement which has grown in importance for out-of-distribution detection, active learning, and ensemble diversity. The anchoring technique is novel enough to potentially inspire new directions in multi-model analysis, particularly for understanding why and when independently trained models diverge despite identical architectures. The paper complements recent work on deep ensemble uncertainty but provides rigorous theoretical guarantees rather than empirical observations, filling a gap in the theoretical understanding of independent model training.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::