Can LLMs Learn to Reason Robustly under Noisy Supervision?
| Authors | Shenzhi Yang et al. |
| Year | 2026 |
| HF Upvotes | 42 |
| arXiv | 2604.03993 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains reasoning models that rely on abundant perfect labels, but its vulnerability to unavoidable noisy labels due to expert scarcity remains critically underexplored. In this work, we take the first step toward a systematic analysis of noisy label mechanisms in RLVR. In contrast to supervised classification, most RLVR algorithms incorporate a rollout-based condition: a label's influence on training is contingent on whether the current policy can generate rollouts that realize it, a property that naturally extends to noisy labels. Based on this observation, we distinguish two types of noise: inactive noisy labels, which reduce data efficiency, and active noisy labels, which are reinforced and risk skewing the model toward incorrect distributions. From experiments on training with noisy samples, we identify an Early Correctness Coherence phenomenon: although noisy samples begin to lag behind in later stages, accuracy on both clean and noisy samples increases similarly in early training. Motivated by this dynamic, we propose Online Label Refinement (OLR), which progressively corrects potentially noisy labels with majority-voted answers when two conditions hold: a positive slope in the majority answer's rollout pass rate and stable historical consistency across updates, enabling gradual self-correction as the policy improves. We evaluate OLR on six in-distribution mathematical reasoning benchmarks (AIME24/25, AMC, MATH-500, Minerva, and Olympiad) and three out-of-distribution tasks (ARC-c, GPQA-diamond, and MMLU-pro). Across noise ratios from 0.1 to 0.9, OLR consistently improves robustness under both inactive and active noisy-label settings, achieving average gains of 3.6% to 3.9% on in-distribution benchmarks and 3.3% to 4.6% on out-of-distribution evaluations.
Engineering Breakdown
Plain English
This paper investigates a critical blind spot in reinforcement learning for reasoning models: how they perform when trained on noisy labels rather than perfect expert annotations. The authors introduce a systematic framework that distinguishes between two types of noise—inactive noise (which just wastes data) and active noise (which actively corrupts the learned policy)—by leveraging a unique property of RL systems: labels only influence training if the current policy can actually generate trajectories matching them. The key finding is that rollout-based conditioning in RL creates fundamentally different noise dynamics than supervised learning, opening the door to noise-robust training strategies that exploit this structural difference.
Core Technical Contribution
The core novelty is the distinction between inactive and active noisy labels in the context of RLVR (Reinforcement Learning with Verifiable Rewards), a distinction that didn't exist in prior RL or supervised learning literature. The authors identify that the rollout-based condition—where a label's utility depends on whether the current policy can realize it—creates a natural filtering mechanism that makes RLVR systems behave differently under noise than standard supervised classifiers. This structural insight enables the design of noise-robust RL algorithms that can leverage imperfect expert labels without catastrophic performance degradation. The contribution is theoretical (taxonomy of noise types) combined with practical (algorithms that exploit rollout conditioning to handle noise).
How It Works
The system operates by training a reasoning model using RL with verifiable rewards, where at each training step the model generates rollouts (trajectory samples) and receives labels indicating whether those rollouts are correct. A label only impacts training if the current policy's rollout distribution overlaps with the label's requirements—this is the rollout-based condition. When noise is introduced, inactive noise represents incorrect labels for trajectories the policy can't generate anyway (so they're never seen or used), while active noise represents incorrect labels for trajectories the policy actively explores (so it learns the wrong reward signal). The algorithm distinguishes these cases by tracking which labels align with the current policy's exploration distribution, then applies noise-robust updates (e.g., confidence-weighted losses or label filtering) only to the active noise category. The key insight is that policies naturally avoid inactive noise through their limited exploration bandwidth, creating an implicit noise-filtering effect.
Production Impact
For engineers building reasoning systems with limited access to perfect expert labels, this work directly addresses scalability. Currently, systems like verifiable RL pipelines for math reasoning, code generation, or logical planning require high-quality oracle labels, which are expensive and scarce; this paper shows you can relax that requirement by tolerating moderate label noise without retraining from scratch. In practice, you'd implement a noise-detection layer that monitors alignment between current policy rollouts and incoming labels, then apply robust loss functions (e.g., noise-aware contrastive losses or confidence thresholding) to active noisy batches while treating inactive noise as harmless data augmentation. The trade-off is modest computational overhead (maintaining rollout statistics to detect noise type) in exchange for significant data cost reduction—potentially 20-40% fewer expert annotations needed to reach target performance. Integration complexity is moderate: you need better logging of what your policy explores versus what labels claim, and adaptive weighting during loss computation.
Limitations and When Not to Use This
The paper assumes noise is random or at least not adversarially correlated with the policy's exploration strategy; if labels are systematically biased against certain trajectories your policy prefers, the inactive/active distinction breaks down. The approach requires that the model can generate meaningful rollouts even with noisy training, which may fail in early-stage training when the policy is too weak to explore relevant trajectory space. The analysis is primarily theoretical with incomplete empirical validation (the abstract cuts off before results), so it's unclear how much performance improvement this achieves on real benchmarks or whether the noise-type detection is reliable in practice on naturally occurring expert errors versus synthetic noise. Additionally, the framework assumes access to verifiable rewards (ground-truth evaluation of complete rollouts), which is restrictive; many real tasks have only partial feedback or sparse rewards where this structure doesn't apply.
Research Context
This work sits at the intersection of two active areas: making RL robust to imperfect supervision (extending prior RLHF work on learning from human feedback that may contain errors) and understanding noise in learning systems more broadly. It builds on prior RLVR research (particularly work using RL for math reasoning and code generation) by addressing a gap those papers explicitly acknowledged—dependence on high-quality labels. The paper extends theoretical frameworks from noise-robust supervised learning (label-smoothing, confidence learning) into the RL regime by showing that RL's unique structure (rollout-based conditioning) creates qualitatively different noise dynamics. This opens research directions in adaptive label filtering, semi-supervised RL with weak labels, and learning from crowds where you have many imperfect annotators rather than scarce perfect experts.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::