Beyond Distribution Sharpening: The Importance of Task Rewards
| Authors | Sarthak Mittal et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2604.16259 |
| Download | |
| Categories | cs.LG, cs.AI |
Abstract
Frontier models have demonstrated exceptional capabilities following the integration of task-reward-based reinforcement learning (RL) into their training pipelines, enabling systems to evolve from pure reasoning models into sophisticated agents. However, debate persists regarding whether RL genuinely instills new skills within a base model or merely sharpens its existing distribution to elicit latent capabilities. To address this dichotomy, we present an explicit comparison between distribution sharpening and task-reward-based learning, utilizing RL as a tool to implement both paradigms. Our analysis reveals the inherent limitations of distribution sharpening, demonstrating from first principles how and why the optima can be unfavorable and the approach fundamentally unstable. Furthermore, our experiments using Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct and Qwen3-4B-Instruct-2507 on math datasets confirm that sharpening yields limited gains, whereas incorporating task-based reward signal can greatly help achieve robust performance improvements and stable learning.
Engineering Breakdown
Plain English
This paper challenges the prevailing assumption that reinforcement learning in frontier models works primarily by sharpening the base model's existing distribution to reveal latent capabilities. Instead, the authors present a rigorous first-principles analysis comparing distribution sharpening against task-reward-based learning, demonstrating that pure distribution sharpening has fundamental limitations and instability problems. Using RL as a testbed to implement both paradigms, they show empirically and theoretically that task rewards genuinely teach models new skills rather than merely extracting existing ones. The work provides concrete evidence that the mechanism enabling modern frontier models to evolve from pure reasoners into capable agents involves actual skill acquisition, not just distribution manipulation.
Core Technical Contribution
The core contribution is a formal theoretical and empirical distinction between two mechanistically different ways RL affects model behavior: distribution sharpening (concentrating probability mass on the model's highest-confidence outputs) versus task-reward-based learning (genuinely expanding the model's capability set). The authors derive first-principles mathematical arguments showing why distribution sharpening leads to unfavorable optima and inherent instability, providing bounds on performance degradation. They implement both paradigms explicitly within an RL framework and measure their effects separately, isolating the contribution of task rewards from distribution effects. This explicit separation methodology is novel and enables direct measurement of whether capability gains come from new skills or merely from better extraction of existing ones.
How It Works
The approach implements two distinct RL training regimes on the same base model: one optimizing for distribution sharpening alone (concentrating probability on the model's current highest-likelihood outputs) and one optimizing for task rewards that measure actual task success. Distribution sharpening is implemented by maximizing the probability of the model's own top-k predictions, creating a self-reinforcing loop that tightens the output distribution without expanding capabilities. Task-reward-based learning instead uses RL to optimize an external reward function measuring whether the model's outputs actually solve the given task, independent of the original distribution. By comparing downstream performance, sample efficiency, and stability metrics between these two approaches on identical base models and datasets, the authors isolate the causal effect of genuine task learning. The paper includes theoretical analysis of the loss landscape and convergence properties of each approach, explaining why distribution sharpening leads to mode collapse and suboptimal solutions.
Production Impact
For engineers building production AI systems, this work directly informs the design of RL fine-tuning pipelines for frontier models. If distribution sharpening is insufficient, teams should prioritize task-specific reward signals and ensure RL training objectives measure actual task success rather than confidence concentration or KL divergence from base models. This suggests that model scaling alone without careful reward engineering may hit capability ceilings, and that investment in reward modeling and task-specific metrics yields genuine capability gains rather than cosmetic improvements. The stability analysis has practical implications: distribution-sharpening approaches may require careful hyperparameter tuning and early stopping to avoid collapse, whereas task-reward methods have more stable convergence properties at scale. Production deployments should measure both accuracy improvements and distribution shift: true capability gains will show consistent improvements across diverse prompts and evaluation sets, while sharpening-only effects will concentrate gains in high-confidence regions and may degrade out-of-distribution robustness.
Limitations and When Not to Use This
The paper does not fully address how to design task rewards that are aligned with end-user needs and don't themselves introduce gaming or specification problems—the assumption that good task rewards exist is taken as given. It focuses on discrete task-completion metrics and may not directly apply to open-ended generation tasks (creative writing, research assistance) where task success is harder to define formally. The theoretical bounds and instability results assume specific distributional properties and reward structures that may not hold for all model architectures or domains; generalization to multimodal or extremely large-scale models (1T+ parameters) remains unexplored. Additionally, the paper does not deeply investigate the sample efficiency trade-off: while task-reward-based learning teaches new skills, it may require substantially more human feedback or environment interaction than distribution sharpening, creating a practical cost calculation that varies by domain.
Research Context
This work directly engages with debates within the reinforcement learning for language models community about the mechanisms driving frontier model improvements, building on prior work in RLHF (Reinforcement Learning from Human Feedback) and process reward modeling. It extends and formalizes critiques of KL-divergence-heavy training approaches and provides rigorous grounding for intuitions from DeepSeek-R1, o1, and other process-reward models that emphasize task-specific learning. The paper contributes to the broader research direction of mechanistic understanding of RL in LLMs, opening new questions about reward design, capability emergence, and the relationship between model scale and skill acquisition. It also informs the theoretical foundations of AI safety work by clarifying whether capability increases come from genuine learning (which enables better interpretability and control) or distribution artifacts (which may mask brittle reasoning).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::