ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning
| Authors | Zihan Lin et al. |
| Year | 2026 |
| HF Upvotes | 5 |
| arXiv | 2605.00380 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4% in Avg@16 and 7.0% in Pass@128. Code is available at https://github.com/1229095296/ResRL.git.
Engineering Breakdown
Plain English
This paper addresses a fundamental trade-off in reinforcement learning for large language models: when you train models to maximize rewards for correct reasoning, they tend to produce repetitive, low-diversity outputs. The authors propose Residual Reinforcement Learning (ResRL), which uses negative sample projection to decouple overlapping semantic patterns between good and bad responses, allowing the model to learn strong reasoning without collapsing into reward-chasing monotony. The key insight is linking a theoretical concept called Lazy Likelihood Displacement (LLD) to gradient interference patterns during training, then deriving a practical single-forward proxy that makes the approach computationally tractable. This enables better reasoning ability while maintaining the semantic diversity needed for realistic language generation.
Core Technical Contribution
The core novelty is the negative sample projection mechanism that explicitly decouples semantic distributions shared between positive (correct) and negative (incorrect) responses during RL training. Rather than simply upweighting penalties on negative samples like prior work (NSR), ResRL surgically separates the overlapping feature representations, preventing the model from overshooting toward a narrow reward-optimized mode. Theoretically, the authors establish a connection between Lazy Likelihood Displacement and gradient interference in the negative-positive head (the final classification layer), showing how this interference causes diversity collapse. They then derive a computationally efficient single-forward proxy to estimate and mitigate this interference without requiring multiple forward passes, making the approach practical for large models.
How It Works
The method operates in three key stages during RL training. First, when the model generates a response, it computes both a reward signal (from a verifier) and representations from positive and negative samples. Second, ResRL projects negative sample representations to remove components that align with positive sample distributions, effectively creating orthogonal feature spaces for correct vs. incorrect reasoning paths. Third, the single-forward proxy estimates gradient interference by measuring how updates to negative-sample gradients would conflict with positive-sample gradients, then adjusts the loss function accordingly without computing the full interference matrix explicitly. During generation, the model learns to navigate between the positive reward landscape (strong reasoning) and the decoupled negative landscape (diverse failure modes), allowing it to generalize reasoning without memorizing a single 'correct' response style.
Production Impact
For teams building reasoning-enhanced LLM systems (like code generation, math solving, or verification tasks), ResRL directly addresses the 'reward hacking' problem where fine-tuned models produce syntactically correct but semantically repetitive outputs. In production, this means you can train verifiable reward signals (e.g., test case correctness) without sacrificing output diversity needed for downstream tasks like ensemble decoding or user-facing variety. The computational overhead is minimal — a single forward pass per training step instead of multiple passes for baseline NSR — so integration into existing RL pipelines requires only modest changes to the loss computation. However, you'll need well-calibrated verifier rewards and sufficient negative examples in your training data; weak reward signals or imbalanced pos/neg ratios will undermine the decoupling mechanism. This approach is most valuable when you have clear correctness signals (test-driven domains) rather than vague preference judgments.
Limitations and When Not to Use This
The paper assumes access to high-quality negative samples that meaningfully share semantic space with positive samples; in domains where negative examples are just noise rather than near-misses, the projection mechanism may not effectively decouple anything. The theoretical analysis relies on assumptions about gradient geometry (lazy training regime, specific loss landscapes) that may not hold across all model scales or training schedules — the single-forward proxy is still an approximation, and its quality degrades if these assumptions are violated. The method requires defining what 'semantic similarity' means operationally (likely through some feature-space distance), and this definition choice is not thoroughly explored in the abstract; poor choices here could reintroduce the original diversity problem. Finally, the paper doesn't address how ResRL scales to extremely long reasoning chains or multi-step verification problems where positive/negative trajectories may diverge substantially, leaving open whether the approach works for complex planning tasks.
Research Context
This work builds on the RLHF/RLVR paradigm (using verifiable rewards to improve LLM reasoning) and extends recent work on negative sample weighting strategies by adding explicit representation geometry management. It positions itself as an improvement over Negative Sample Reinforcement (NSR) approaches that simply upweight negative penalties but risk suppressing useful shared structure. The paper contributes to the broader goal of making RL training for LLMs more reliable and diverse — addressing a known problem in systems like DeepSeek-R1 and other reasoning models where over-optimization leads to repetitive outputs. It opens research directions around understanding and controlling gradient interference in multi-objective RL and suggests that representation geometry is a crucial (previously overlooked) lever for controlling training dynamics in large-scale language model fine-tuning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::