Prediction-powered Inference by Mixture of Experts
| Authors | Yanwu Gu et al. |
| Year | 2026 |
| Field | Statistics / ML |
| arXiv | 2604.27892 |
| Download | |
| Categories | stat.ML, cs.LG, stat.AP |
Abstract
The rapidly expanding artificial intelligence (AI) industry has produced diverse yet powerful prediction tools, each with its own network architecture, training strategy, data-processing pipeline, and domain-specific strengths. These tools create new opportunities for semi-supervised inference, in which labeled data are limited and expensive to obtain, whereas unlabeled data are abundant and widely available. Given a collection of predictors, we treat them as a mixture of experts (MOE) and introduce an MOE-powered semi-supervised inference framework built upon prediction-powered inference (PPI). Motivated by the variance reduction principle underlying PPI, the proposed framework seeks the mixture of experts that achieves the smallest possible variance. Compared with standard PPI, the MOE-powered inference framework adapts to the unknown performance of individual predictors, benefits from their collective predictive power, and enjoys a best-expert guarantee. The framework is flexible and applies to mean estimation, linear regression, quantile estimation, and general M-estimation. We develop non-asymptotic theory for the MOE-powered inference framework and establish upper bounds on the coverage error of the resulting confidence intervals. Numerical experiments demonstrate the practical effectiveness of MOE-powered inference and corroborate our theoretical findings.
Engineering Breakdown
Plain English
This paper addresses the practical problem of building inference systems when labeled data is scarce but unlabeled data is plentiful. The authors propose a framework that treats multiple pre-trained AI models (like different neural networks or classifiers) as a mixture of experts and combines their predictions intelligently to improve semi-supervised inference. They build on prediction-powered inference (PPI), a statistical technique that leverages model predictions on unlabeled data to reduce variance in parameter estimation. The key innovation is automatically learning the optimal mixture weights for these diverse experts to achieve the smallest possible variance, rather than using a fixed weighting scheme or training a single model from scratch.
Core Technical Contribution
The core technical novelty is extending prediction-powered inference from single predictors to multiple heterogeneous predictors through a principled mixture-of-experts framework. Rather than treating expert selection or combination as a separate problem, the authors directly optimize mixture weights to minimize the asymptotic variance of statistical estimators—this is a variance minimization objective grounded in statistical theory rather than task-specific loss functions. This creates a mathematically principled way to combine diverse models (which may have different architectures, training data, or domains) without retraining or fine-tuning them. The approach is fundamentally different from standard ensemble methods because it targets statistical efficiency in semi-supervised settings rather than classification accuracy.
How It Works
The framework operates as follows: given a collection of pre-trained predictor models and a small labeled dataset plus a large unlabeled dataset, the system first computes predictions from each expert on both labeled and unlabeled data. These predictions are then aggregated through a learned mixture weighting scheme, where each expert receives a weight that contributes to the final combined prediction. The statistical procedure combines the labeled data directly with the prediction-powered estimates from the unlabeled data, using a pseudo-labeling or importance-weighting mechanism. The mixture weights are optimized to minimize the asymptotic variance of the resulting statistical estimator—this is solved via a convex optimization problem that requires only the predictions themselves, not access to model internals. The final output is a parameter estimate (e.g., class probabilities, regression coefficients) with provably reduced variance compared to using any single expert or standard PPI alone.
Production Impact
For engineers building real systems, this approach directly addresses a common bottleneck: you have multiple off-the-shelf models (pretrained language models, domain-specific classifiers, vendor APIs) and limited labeled data for your specific task. Instead of expensive retraining or complex model selection pipelines, you can immediately combine these models through the MOE-PPI framework with minimal engineering overhead—just feed predictions into the variance-minimizing mixer. This is particularly valuable in semi-supervised scenarios like content moderation, medical imaging, or anomaly detection where unlabeled data is cheap but labeling is expensive. The computational cost is low (mixture weight optimization is convex) and the approach is model-agnostic, so it integrates cleanly with existing prediction systems. However, you need sufficient unlabeled data (the framework relies on large-scale statistics) and the quality of each expert matters—garbage experts will still degrade the mixture despite optimal weighting.
Limitations and When Not to Use This
The framework assumes that the collection of experts is reasonably diverse and that unlabeled data comes from the same distribution as the test set—if all experts are highly correlated or the unlabeled data distribution shifts, variance reduction gains diminish significantly. The paper does not deeply address how to select which experts to include, potentially wasting compute if you blindly add poor-quality predictors. Theoretical variance guarantees require asymptotic conditions (large unlabeled dataset size) that may not hold with finite samples, and the method provides no mechanism for detecting or handling adversarially-chosen or systematically-biased experts. Additionally, the framework targets statistical parameter estimation rather than direct predictive accuracy, so performance on tasks where you care most about raw classification or regression error (rather than calibrated confidence intervals) may not be optimal.
Research Context
This work builds directly on prediction-powered inference (PPI), a recent statistical framework by Angelopoulos et al. that combines labeled and unlabeled data for improved estimation under distribution shift. The mixture-of-experts extension is motivated by the explosion of diverse pre-trained models in industry and academia, creating a practical need to combine specialized predictors. It connects to classical ensemble learning theory and mixture-of-experts literature but applies them in the semi-supervised, variance-minimization setting rather than standard supervised learning. The approach opens avenues for studying multi-expert semi-supervised inference across domains (NLP, vision, tabular data) and potentially extending to active learning scenarios where you can strategically choose which new samples to label.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::