The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation
| Authors | Jiaxin Zhang et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.16830 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
On-policy distillation (OPD) is an increasingly important paradigm for post-training language models. However, we identify a pervasive Scaling Law of Miscalibration: while OPD effectively improves task accuracy, it systematically traps models in severe overconfidence. We trace this failure to an information mismatch: teacher supervision is formed under privileged context available during training, whereas the deployed model must report confidence using only deployment-time information. We formalize this perspective theoretically, showing that teacher-conditioned success is generally not a valid target for deployment-time confidence and that helpful privileged context induces entropy collapse and a systematic optimism bias. To address this, we propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts, replaces self-reported confidence with this student-grounded target, and distills the revised response through the same self-distillation pipeline. Experiments across various models and domains show that CaOPD achieves Pareto-optimal calibration while maintaining competitive capability, generalizing robustly under out-of-distribution and continual learning. Our findings highlight that capability distillation does not imply calibrated confidence, and that confidence should be treated as an essential objective in post-training. Code: https://github.com/SalesforceAIResearch/CaOPD
Engineering Breakdown
Plain English
This paper identifies a critical failure mode in on-policy distillation (OPD), a technique used to improve language models after initial training: while OPD successfully boosts task accuracy, it creates severely overconfident models that don't know when they're wrong. The authors discover this happens because teachers supervise models using information available during training (like full context), but deployed models must express confidence using only what's available at deployment time—a fundamental mismatch. They formalize this theoretically, proving that teacher-conditioned targets systematically cause entropy collapse and optimism bias. The paper proposes CaOP, a calibration-aware OPD framework that addresses this by aligning training-time supervision with deployment-time information constraints.
Core Technical Contribution
The core contribution is identifying and formalizing the 'Scaling Law of Miscalibration'—a systematic phenomenon where larger, better-performing OPD models become increasingly overconfident. Rather than treating calibration as a post-hoc problem, the authors prove theoretically that privileged context during training (information the teacher sees but the deployed model won't have) is fundamentally incompatible with deployment-time confidence estimation. They reframe the problem as an information mismatch between training and deployment, not a model capacity issue. The CaOP framework operationalizes this insight by constraining teacher supervision to only use information available at deployment, which is novel because prior OPD work ignored this constraint entirely.
How It Works
The mechanism works in three stages. First, during training, the teacher model generates supervision signal—but critically, CaOP restricts this to only using features/context that will be available when the model is deployed in production, eliminating privileged information leakage. Second, the student model learns from this constrained supervision, training confidence estimates calibrated to deployment-time information availability rather than training-time information. Third, at inference time, the model's uncertainty estimates remain valid because they were trained under the same information constraints it faces in production. The key algorithmic insight is that entropy collapse happens because training entropy is artificially high (due to privileged context), so the model learns to over-compress uncertainty; by matching entropies between training and deployment, the model learns more honest confidence. The framework likely modifies the distillation loss to include a calibration term that penalizes confidence estimates not aligned with deployment-time information availability.
Production Impact
For engineers deploying language models in high-stakes settings (healthcare, finance, autonomous systems), this addresses a critical safety gap: overconfident models don't trigger uncertainty-based fallback mechanisms like human review or retrieval-augmented generation. Adopting CaOP would require auditing what information your teacher model uses during distillation and restricting it to deployment-available features—this might mean removing access to full conversation history, external knowledge bases, or other privileged signals during training. Latency impact is minimal since it modifies training, not inference. The main trade-off is potentially lower peak task accuracy (since you're removing useful training signals) in exchange for calibrated confidence, which enables safer deployment and better-informed downstream decisions. Integration is straightforward if you're already using OPD—it's primarily a loss function and data selection change during the distillation phase.
Limitations and When Not to Use This
The paper doesn't address scenarios where knowing what information is 'privileged' is itself unclear or where the boundary between training and deployment information shifts (e.g., continual learning systems or federated settings where distribution changes post-deployment). It assumes you can cleanly separate training-time vs. deployment-time information, which may not hold in practice—real systems have nuanced feature engineering where some signals leak across boundaries. The approach may suffer in domains where privileged information is abundant and hard to avoid (e.g., multimodal models with text + image where you can't train with only text). The paper doesn't fully explore the accuracy-calibration trade-off curve or provide guidance on how much accuracy loss is acceptable for good calibration in different domains. Additionally, CaOP likely only solves miscalibration from the information mismatch, not from other sources like model capacity mismatch or distributional shift at deployment time.
Research Context
This work sits in the intersection of knowledge distillation (a mature field) and model calibration (increasingly important for safety-critical ML). It builds on recent OPD work that shows distillation can continue improving models post-training, but identifies a blind spot—prior work optimized for accuracy without considering the information asymmetry problem. The paper contributes to the broader trend of treating calibration as a first-class concern in LLM training, alongside recent work on uncertainty quantification and confidence estimation in large models. The theoretical framework connecting privileged information to entropy collapse may inspire similar analyses in other domains like federated learning or continual learning where information availability differs across time or sites.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::