Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
| Authors | Yinghui He et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2604.12002 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization.
Engineering Breakdown
Plain English
This paper introduces Self-Distillation Zero (SD-Zero), a post-training method that improves language model performance using only binary rewards without requiring external teachers or high-quality demonstration data. The core innovation is training a single model to act in two roles: a Generator that produces initial responses, and a Reviser that improves those responses by conditioning on the Generator's output and its binary reward signal. SD-Zero achieves better training sample efficiency than reinforcement learning approaches while eliminating the costly data collection burden of distillation-based methods. The method directly addresses a practical pain point in verifiable AI systems where obtaining dense supervision signals is either expensive, time-consuming, or simply unavailable.
Core Technical Contribution
The key novelty is the dual-role architecture where a single model learns to both generate and iteratively revise its own outputs using only sparse binary reward signals. Unlike traditional distillation that requires external teacher models or curated demonstrations, SD-Zero bootstraps improvement by having the model learn from its own generation-revision cycles conditioned on reward feedback. This is fundamentally different from pure RL approaches that struggle with sparse rewards and from distillation approaches that require expensive data collection. The self-directed improvement mechanism enables the model to discover high-quality refinements without external supervision, making the training process more autonomous and scalable.
How It Works
The SD-Zero architecture operates through a sequence of well-defined steps: (1) The Generator component produces an initial response token sequence from the input prompt using standard autoregressive generation. (2) This response is evaluated by a binary reward signal (e.g., pass/fail, correct/incorrect, or thumbs-up/thumbs-down). (3) The Reviser component then conditions on three inputs—the original prompt, the Generator's output, and the binary reward signal—to produce an improved response. (4) Both Generator and Reviser are implemented as a single model with different conditioning contexts, allowing parameter sharing and efficient training. (5) The training objective optimizes both components to maximize the probability of generating high-quality responses, with the Reviser learning to correct or improve upon the Generator's mistakes. (6) This creates a self-improving loop where the model learns revision strategies purely from the binary feedback signal, without access to reference solutions or external teacher logits.
Production Impact
For teams building verifiable AI systems, SD-Zero eliminates a major bottleneck: the need to collect expensive ground-truth labels or maintain external teacher models for distillation. In production, this means you can deploy post-training on your own models using only lightweight binary feedback signals (e.g., user ratings, automated checker outputs, or pass-fail test results), significantly reducing time-to-improvement. The single-model architecture also reduces inference cost—you can use the same weights for both generation and revision phases, avoiding the multi-model orchestration complexity of traditional distillation. The main trade-off is that this method requires careful design of the binary reward signal and assumes the model has sufficient capacity to learn both generation and revision; for tasks where reward signals are noisy or poorly calibrated, performance gains may be limited. Integration-wise, you'd need to modify your inference pipeline to optionally invoke the Reviser based on an initial generation, adding minimal latency overhead compared to multi-model approaches.
Limitations and When Not to Use This
The paper's abstract indicates this method relies on binary reward signals, which can be coarse-grained and may not capture nuanced quality differences—this could limit performance on tasks requiring fine-grained optimization. The approach assumes a single model can effectively learn two distinct roles (generation and revision) without interference, which may not hold for all architectures or task distributions. The method hasn't yet demonstrated effectiveness on extremely complex reasoning tasks where multiple refinement iterations might be needed, and the paper doesn't clarify how to handle cases where even the Reviser fails to improve upon the Generator. Additionally, there's no discussion of failure modes when the binary reward signal is misaligned with the actual quality metric, which is a common problem in practice; you'd need strong validation before deploying in high-stakes applications.
Research Context
This work sits at the intersection of reinforcement learning (specifically preference-based RL like RLHF) and knowledge distillation, two dominant post-training paradigms for large language models. It extends beyond prior binary-reward RL methods by introducing the structured revision mechanism and beyond distillation by eliminating external supervision requirements. The research builds on intuitions from chain-of-thought refinement and iterative editing but formalizes these as a learnable two-role framework. This direction opens up new research into self-supervised iterative improvement and opens questions about scaling these methods to longer reasoning chains and more complex tasks, potentially influencing how we approach post-training for future frontier models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::