Self-Distilled Agentic Reinforcement Learning
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 84 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Zhengxi Lu et al. |
| Year | 2026 |
| HF Upvotes | 84 |
| arXiv | 2605.15155 |
| Download | |
| Code | https://github.com/ZJU-REAL/SDAR |
Abstract
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.
Engineering Breakdown
Plain English
This paper tackles a fundamental problem in RL-based LLM agent training: trajectory-level rewards are too sparse and coarse-grained to effectively guide long-horizon multi-turn interactions. The authors propose SDAR, which combines on-policy self-distillation (using a teacher model with privileged information to provide dense token-level supervision) with standard RL, gating the distillation signal as an auxiliary objective rather than forcing it as primary, and handling asymmetric cases where the teacher model might reject actions due to imperfect skill retrieval.
Key Engineering Insight
The critical engineering insight is treating dense supervision (self-distillation) and sparse rewards (RL) as complementary signals with explicit gating rather than conflicting objectives—this avoids training instability when the teacher and agent disagree on multi-turn trajectories, which is the core failure mode of naive combinations.
Why It Matters for Engineers
Production LLM agents fail on long, multi-step tasks because reward signals only arrive at the end of trajectories, leaving intermediate steps unguided. SDAR directly addresses this by injecting per-token guidance from a privileged teacher during training, potentially improving sample efficiency and reliability of agentic systems without requiring architectural changes to deployed models.
Research Context
Prior work showed on-policy self-distillation works well for single-step or short-horizon tasks, but multi-turn agents suffer from compounding instability when teacher and student disagree. This paper advances the field by proving you can combine RL and distillation safely via gating, which enables denser supervision signals for the long-horizon agentic tasks that are increasingly central to LLM post-training pipelines.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::