Self-Distilled RLVR
| Authors | Chenxu Yang et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.03128 |
| Download | |
| Categories | cs.LG, cs.CL |
Abstract
On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose \textbf{RLSD} (\textbf{RL}VR with \textbf{S}elf-\textbf{D}istillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
Engineering Breakdown
Plain English
This paper addresses a critical instability problem in on-policy self-distillation (OPSD) for large language models, where a single model acts as both teacher and student using privileged information like reference answers. The authors found that relying solely on learning signals from the privileged teacher causes severe information leakage and unstable training over long horizons. They propose an optimal combination strategy that balances signals from both the verifiable environment rewards (RLVR) and the privileged teacher distillation, enabling stable and effective self-evolution of language models without requiring larger teacher models.
Core Technical Contribution
The key novelty is identifying and solving the information leakage problem in on-policy self-distillation by proving that purely teacher-derived signals are insufficient for stable long-term training. The authors develop a principled approach to optimally blend reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), showing that this hybrid signal combination prevents distribution collapse and maintains training stability. This contrasts sharply with prior OPSD work that assumed teacher signals alone were sufficient, and with pure RLVR approaches that ignore the dense pedagogical value of privileged information. The contribution is both empirical (demonstrating instability) and algorithmic (providing the optimal mixing strategy).
How It Works
The system operates by running a language model that generates trajectories (sequences of tokens), then evaluates these trajectories using two independent signal sources. The first source is sparse verifiable rewards from the environment (RLVR)—hard signals like whether a solution is correct. The second source is dense distillation signals from the same model acting as teacher, using privileged information like reference answers to provide fine-grained trajectory-level feedback. The training process jointly optimizes on both signals, with the key innovation being the identification of an optimal weighting coefficient (or curriculum schedule) that governs how much to trust each signal source at each training step. Early training emphasizes RLVR signals to establish a stable base distribution, while later stages can leverage more teacher distillation as the model becomes robust. The paper likely derives this optimal ratio through theoretical analysis or empirical search, preventing the model from drifting away from verifiable ground truth.
Production Impact
For teams building LLM systems with verification capability, this approach offers a practical path to faster training without requiring expensive larger teacher models—the model distills from itself. Instead of maintaining separate large teacher models (adding 2-3x compute cost), you run a single model generating data and receiving dual signals, reducing inference overhead. The stability guarantees are critical: production systems need predictable training dynamics and convergence; pure OPSD's information leakage would cause performance cliffs or divergence mid-training, requiring costly intervention or rollback. Integration is straightforward: existing RLVR pipelines add a privileged information channel (reference answers) and modify the loss function to weight both signal sources, typically adding 10-20% computational overhead during training. The main trade-off is that you need access to privileged information (correct answers) during training—this works for code generation, math, QA, and constrained domains but not for open-ended tasks without ground truth.
Limitations and When Not to Use This
The paper's approach assumes reliable privileged information is available at training time, which is unavailable or expensive to obtain for many real-world tasks like creative writing, summarization, or dialogue. The optimal blending ratio discovered for one domain (likely math or code from the authors' framing) may not transfer to different tasks, requiring re-tuning per application. The paper doesn't address computational complexity or training time comparisons against baseline OPSD or pure RLVR—if the hybrid approach requires careful scheduling and extensive tuning, it may not be practical at scale. Long-term stability is claimed but the paper likely only evaluates up to a fixed training horizon; truly unbounded training with distribution drift over weeks or months remains unvalidated. Additionally, the information leakage analysis is probably specific to the models and domains tested; generalization to different architectures (mixture-of-experts, sparse models) or multimodal settings is unclear.
Research Context
This work extends the on-policy distillation (OPD) paradigm that has dominated recent LLM training (following scaling laws and RLHF-style approaches), specifically building on on-policy self-distillation methods that emerged as a way to remove dependency on external teacher models. It bridges two communities: the reinforcement learning with verifiable rewards (RLVR) camp focused on hard signals for constrained domains, and the knowledge distillation camp focused on dense pedagogical signals. The paper likely improves benchmarks in formal reasoning (MATH, code generation tasks like HumanEval or LeetCode) where both verification and reference answers are available. This opens a research direction toward hybrid signal learning, suggesting future work could explore multi-source reward combination, dynamic weighting schedules, or theoretical foundations for signal fusion in self-supervised learning systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::