Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means
| Authors | Raphaël Razafindralambo et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2603.04204 |
| Download | |
| Categories | stat.ML, cs.CV, cs.LG, stat.ME |
Abstract
Density aggregation is a central problem in machine learning, for instance when combining predictions from a Deep Ensemble. The choice of aggregation remains an open question with two commonly proposed approaches being linear pooling (probability averaging) and geometric pooling (logit averaging). In this work, we address this question by studying the normalized generalized mean of order r \in \mathbb{R} \cup \{-\infty,+\infty\} through the lens of log-likelihood, the standard evaluation criterion in machine learning. This provides a unifying aggregation formalism and shows different optimal configurations for different situations. We show that the regime is the only range ensuring systematic improvements relative to individual distributions, thereby providing a principled justification for the reliability and widespread practical use of linear () and geometric () pooling. In contrast, we show that aggregation rules with may fail to provide consistent gains with explicit counterexamples. Finally, we corroborate our theoretical findings with empirical evaluations using Deep Ensembles on image and text classification benchmarks.
Engineering Breakdown
Plain English
This paper tackles the problem of how to best combine predictions from multiple models (like Deep Ensembles) by studying different aggregation methods through a unified mathematical framework. The authors analyze the normalized generalized mean of order r, which encompasses both linear pooling (averaging probabilities) and geometric pooling (averaging logits) as special cases, evaluated using log-likelihood. Their key finding is that values of r between 0 and 1 consistently outperform individual model predictions, providing principled guidance on which aggregation strategy to use in different scenarios. This unifies what was previously treated as separate heuristics into a single theoretical framework with concrete optimal configurations.
Core Technical Contribution
The paper's core novelty is reformulating ensemble density aggregation through the lens of the normalized generalized mean parameterized by r, creating a unifying framework that encompasses both linear and geometric pooling as special cases. Rather than treating these as disconnected techniques, the authors show they sit on a continuous spectrum and can be analyzed through a single mathematical lens: log-likelihood optimization. The key discovery is that the interval r ∈ [0,1] represents the only range that guarantees systematic improvements over individual distributions, providing an actionable principle for practitioners. This shifts ensemble aggregation from an ad-hoc choice to a theoretically grounded problem with well-defined optimal regimes for different situations.
How It Works
The approach starts with an input ensemble of probability distributions from multiple models. The normalized generalized mean of order r is applied as an aggregation operator, where r acts as a tuning parameter: r=1 recovers linear pooling (arithmetic mean of probabilities), r→0 approaches geometric pooling (logit averaging), and r=-∞ gives the pointwise maximum. For each configuration, the aggregated distribution is evaluated using log-likelihood—the standard evaluation metric in machine learning—to assess how well it predicts held-out data. The authors systematically sweep across the space of r values and document which regions deliver better log-likelihood than any individual model in the ensemble. By analyzing this landscape, they identify that r ∈ [0,1] is the only range where improvements are guaranteed, enabling practitioners to select an aggregation method based on their specific dataset characteristics rather than guessing between linear and geometric pooling.
Production Impact
For engineers deploying ensemble models, this provides concrete guidance on aggregation strategy selection during inference. Instead of defaulting to probability averaging (linear pooling) or logit averaging (geometric pooling) by convention, practitioners can now empirically validate which r value in [0,1] optimizes log-likelihood on their validation set, then apply that same aggregation at test time. This is computationally cheap—aggregation happens post-inference and requires only a single hyperparameter sweep during validation—but can yield measurable improvements in model calibration and predictive accuracy. For production systems using Deep Ensembles, this translates directly into better uncertainty estimates with minimal latency overhead; the trade-off is negligible (aggregation is O(n) in ensemble size) while the benefit compounds across millions of predictions. For systems already logging ensemble member probabilities, adopting this requires only a tuning script and a single parameter change in the aggregation logic.
Limitations and When Not to Use This
The paper's analysis assumes access to ground-truth labels for validation, which may not be available in real-time prediction scenarios; practitioners need historical data or periodic human feedback to determine the optimal r for their distribution. The framework assumes that the ensemble members are independent or nearly so—if members are highly correlated (e.g., all fine-tuned from the same base model), the improvements may be less pronounced than the theory predicts. The paper focuses on log-likelihood as the evaluation metric; if your production objective is F1-score, AUC, or calibration error, the optimal r may differ and require separate validation. Additionally, the approach requires that all ensemble members output calibrated probability estimates; if some members are poorly calibrated, the aggregation framework cannot fix that underlying issue and may propagate or amplify miscalibration.
Research Context
This work builds on a long tradition of ensemble learning and probabilistic model combination, sitting at the intersection of Bayesian model averaging and ensemble methods literature. It advances prior work that studied linear vs. geometric pooling as disconnected techniques by providing a unified mathematical framework that reveals them as endpoints of a continuous space. The paper contributes to the broader push for principled, theoretically grounded ensemble methods in deep learning, relevant to recent work on Deep Ensembles and their role in uncertainty quantification. This opens research directions on how aggregation regimes vary with ensemble diversity, out-of-distribution detection, and whether the optimal r exhibits useful patterns across different domains or task structures.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::