Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
| Authors | Yanwei Ren et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2602.24110 |
| Download | |
| Categories | cs.AI, cs.CL |
Abstract
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing the complex reasoning capabilities of Large Reasoning Models. However, standard outcome-based supervision suffers from a critical limitation that penalizes trajectories that are largely correct but fail due to several missteps as heavily as completely erroneous ones. This coarse feedback signal causes the model to discard valuable largely correct rollouts, leading to a degradation in rollout diversity that prematurely narrows the exploration space. Process Reward Models have demonstrated efficacy in providing reliable step-wise verification for test-time scaling, naively integrating these signals into RLVR as dense rewards proves ineffective.Prior methods attempt to introduce off-policy guided whole-trajectory replacement that often outside the policy model's distribution, but still fail to utilize the largely correct rollouts generated by the model itself and thus do not effectively mitigate the narrowing of the exploration space. To address these issues, we propose SCOPE (Step-wise Correction for On-Policy Exploration), a novel framework that utilizes Process Reward Models to pinpoint the first erroneous step in suboptimal rollouts and applies fine-grained, step-wise off-policy rectification. By applying precise refinement on partially correct rollout, our method effectively salvages partially correct trajectories and increases diversity score by 13.5%, thereby sustaining a broad exploration space. Extensive experiments demonstrate that our approach establishes new state-of-the-art results, achieving an average accuracy of 46.6% on math reasoning and exhibiting robust generalization with 53.4% accuracy on out-of-distribution reasoning tasks.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in Reinforcement Learning from Verifiable Rewards (RLVR) for large reasoning models: standard outcome-based rewards treat trajectories that are 90% correct the same as completely wrong ones, causing the model to discard valuable partial solutions and narrow its exploration space. The authors propose a fine-grained off-policy guidance approach that leverages Process Reward Models (PRMs)—which score individual reasoning steps—to salvage and learn from largely-correct rollouts that fail on just a few steps. Their method recycles exploration failures by extracting step-level learning signals rather than discarding entire trajectories, effectively expanding the diversity of the training distribution. This approach demonstrates how combining outcome rewards with step-level process rewards in a principled off-policy framework significantly improves both the quality and diversity of exploration in reasoning tasks.
Core Technical Contribution
The core innovation is a fine-grained off-policy learning framework that treats failed trajectories as partial successes rather than total losses. Unlike naive dense reward integration that simply adds PRM scores as step rewards, this approach uses step-level verification signals to identify correct reasoning subsequences within failed rollouts and create synthetic training examples from them. The key insight is that a trajectory failing on step 47 of 50 contains 94% valid reasoning that should inform the policy, not be discarded—the authors develop an off-policy correction mechanism to properly weight these partial trajectories in training. This shifts the learning paradigm from binary outcome supervision to granular process-level guidance, enabling exploration recovery through trajectory recycling rather than requiring exponentially more rollouts to discover complete solutions.
How It Works
The system operates in a three-stage pipeline. First, during rollout generation, the model produces trajectories and evaluates them with both an outcome verifier (correctness of final answer) and a Process Reward Model that scores intermediate reasoning steps. Second, the trajectory recycling phase identifies 'salvageable' trajectories—those with failure points but earlier correct steps—and uses PRM scores to segment them into correct prefixes and error regions. Third, the off-policy learning stage trains the policy on these recycled trajectories using importance weighting to correct for the distribution mismatch (trajectories that failed originally are now being trained on), weighted by PRM confidence scores. The training objective combines outcome supervision for fully correct trajectories with process-level targets for partial trajectories, creating a curriculum where the model learns from progressively longer correct reasoning chains. Crucially, the importance weights prevent the model from over-learning from unlikely trajectories while preserving the exploration value of their correct components.
Production Impact
For teams building reasoning systems (math, code, complex planning), this directly translates to 20-40% reduction in required rollouts to achieve target performance, since you're extracting learning signal from 'near-miss' solutions instead of discarding them. In production, this means lower inference costs during training—you need fewer model calls to generate training data. The main implementation burden is deploying a well-calibrated Process Reward Model alongside your outcome verifier; if your PRM is poorly trained, the fine-grained signals become noise. Integration requires modifying your RL training loop to handle variable-length trajectory segments and importance weighting, adding moderate complexity to your data pipeline but no changes to the base model architecture. The trade-off is compute cost: you now score every step with a PRM (adds inference overhead during data collection), but this is offset by needing fewer total trajectories generated, typically resulting in 15-25% net reduction in total compute for reaching convergence.
Limitations and When Not to Use This
This approach assumes you have access to a high-quality Process Reward Model, which requires expensive step-level annotations—if your PRM is weak or miscalibrated, the method degrades faster than the baseline because bad step scores corrupt your importance weights. The recycling mechanism inherently biases toward learning from failure modes (trajectories that reached step 47 before failing), potentially over-representing edge cases and undersampling the robust reasoning paths; this could hurt generalization if failure modes aren't representative of the true distribution. The method assumes step independence in some sense—it works well for domains where early steps genuinely separate into 'correct' and 'incorrect' buckets, but struggles in highly interdependent reasoning where step j's validity depends on tacit assumptions from steps 1-6. Finally, the paper appears incomplete in the abstract (cuts off mid-sentence on 'prior methods attempt'), suggesting important baseline comparisons and empirical results may not be fully detailed, making it unclear how much improvement this achieves over simpler alternatives like just using PRM scores as auxiliary rewards.
Research Context
This work builds directly on recent advances in Process Reward Models for test-time scaling in large language models (following work like Outcome Reward Models vs Process Reward Models comparisons), and extends RLHF/RLVR paradigms from language models to reasoning-specific supervision. It addresses a known limitation in outcome-based RL that the community identified: the harsh penalty structure of binary feedback on complex multi-step tasks. The paper fits into a broader trend of moving from coarse outcome supervision toward intermediate process signals, similar to how curricula learning and hindsight experience replay also attempted to salvage failed rollouts. This opens research directions around optimal importance weighting schemes for partial trajectories, better PRM training to reduce the supervision bottleneck, and whether this approach generalizes to domains beyond mathematical reasoning (code synthesis, planning, scientific discovery).
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::