SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
| Authors | Binbin Zheng et al. |
| Year | 2026 |
| HF Upvotes | 25 |
| arXiv | 2604.10688 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teacher model, but typically applies this supervision uniformly across all rollouts, ignoring fundamental differences in signal quality. We propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths. For incorrect trajectories, SCOPE performs teacher-perplexity-weighted KL distillation to prioritize instances where the teacher demonstrates genuine corrective capability, while down-weighting unreliable guidance. For correct trajectories, it applies student-perplexity-weighted MLE to concentrate reinforcement on low-confidence samples at the capability boundary rather than over-reinforcing already mastered ones. Both paths employ a group-level normalization to adaptively calibrate weight distributions, accounting for the intrinsic difficulty variance across prompts. Extensive experiments on six reasoning benchmarks show that SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines, demonstrating its consistent effectiveness.
Engineering Breakdown
Plain English
This paper addresses a critical bottleneck in aligning large language models using reinforcement learning: sparse, outcome-level rewards make it extremely hard to assign credit to individual tokens in a sequence. The authors propose SCOPE (Signal-Calibrated On-Policy Distillation Enhancement), which uses a teacher model to provide dense, token-level supervision, but crucially routes training through two different paths depending on trajectory quality. For correct trajectories, it applies standard KL distillation; for incorrect ones, it weights the distillation by teacher perplexity to focus learning on cases where the teacher has high confidence. This dual-path routing adaptively calibrates supervision strength based on signal quality, moving beyond the uniform supervision that prior on-policy distillation methods apply.
Core Technical Contribution
The core innovation is adaptive routing of on-policy rollouts based on trajectory correctness, paired with teacher-perplexity weighting for incorrect trajectories. Prior on-policy distillation (OPD) methods apply identical KL supervision across all samples, treating a rollout where the model confidently goes wrong the same as one where it stumbles on an edge case. SCOPE recognizes that learning signals vary in quality: when a trajectory fails, the teacher's confidence (perplexity) indicates whether it's a case where even the teacher finds the task hard versus where the teacher is certain but the student went astray. The two-path system lets correct trajectories use standard distillation while incorrect ones use weighted distillation, enabling the model to learn more effectively from the distribution of its own mistakes.
How It Works
SCOPE operates as follows: (1) Generate on-policy rollouts from the current model being trained. (2) Evaluate each rollout for correctness using some ground-truth signal. (3) Route correct rollouts to Path A, which applies standard token-level KL divergence between student logits and teacher logits. (4) Route incorrect rollouts to Path B, which computes the teacher's perplexity on those tokens and uses it as a weight in the KL loss — lower teacher perplexity (higher confidence) means higher loss weight, signaling that this is a clear mistake the student should not make. (5) Combine gradients from both paths and update the student model. The key insight is that teacher perplexity serves as a learned signal of supervision quality: if the teacher struggles too (high perplexity), down-weight that signal; if the teacher is confident, up-weight it. This creates a continuous, adaptive weighting scheme rather than binary accept/reject decisions.
Production Impact
For teams training alignment models at scale, SCOPE directly improves sample efficiency during the RL training phase, which is the most compute-intensive stage of LLM alignment. Instead of requiring millions of additional rollouts to achieve the same model quality, the adaptive weighting means each rollout contributes a stronger learning signal, reducing wall-clock training time and GPU hours. In a production pipeline, you would integrate SCOPE between your policy rollout stage and the gradient update stage: collect trajectories, run them through an existing teacher model (reusing one you may already have for evaluation), compute correctness labels, split into two paths, and apply the appropriate loss. The trade-off is modest: teacher inference adds latency to the training loop (typically 10-20% overhead), and you need a pre-trained teacher model with reliable confidence estimates; the compute savings from fewer required rollouts typically outweigh this overhead. This approach scales well to models from 7B to 405B parameters since it doesn't change the model architecture, only the training procedure.
Limitations and When Not to Use This
SCOPE assumes access to accurate correctness labels for each trajectory, which is non-trivial for reasoning tasks where ground truth is expensive or ambiguous (open-ended generation, creative writing, subjective reasoning). The method also depends on the teacher model's perplexity being a good proxy for signal quality, but if the teacher is miscalibrated or overconfident on certain distribution shifts, the weighting scheme will amplify incorrect learning signals rather than calibrating them. The paper appears incomplete (the abstract cuts off mid-sentence), so it's unclear what empirical results, failure modes, or comparisons to baselines are actually reported — critical information for assessing real-world effectiveness. Additionally, the approach is most effective for on-policy training; it's unclear how well the signal quality routing generalizes to offline RL or to settings where you have a diverse replay buffer rather than fresh rollouts, which is common in production systems that need to amortize compute.
Research Context
This work sits at the intersection of two major research threads in LLM alignment: on-policy RL methods (following the success of models like DeepSeek-R1 and OpenAI's o1, which use process rewards and chain-of-thought reasoning) and knowledge distillation from teacher models. SCOPE builds directly on prior work in On-Policy Distillation (OPD), which itself was a response to the credit assignment problem in RL-trained LLMs, and extends it by making the supervision adaptive. The paper likely benchmarks on standard alignment tasks (MATH, code generation, reasoning benchmarks) where token-level credit assignment is particularly important. By introducing perplexity-weighted supervision, the work opens a research direction on adaptive weighting schemes in distillation — future work could explore other metrics beyond perplexity, multi-teacher ensembles, or learned weighting functions that predict signal quality from trajectory features.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::