RAGEN-2: Reasoning Collapse in Agentic RL
| Authors | Zihan Wang et al. |
| Year | 2026 |
| HF Upvotes | 58 |
| arXiv | 2604.06268 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.
Engineering Breakdown
Plain English
This paper identifies a critical failure mode in reinforcement learning training of multi-turn LLM agents called template collapse, where models generate output that appears diverse by entropy metrics but actually ignores input variation. The authors discovered that standard entropy measurements fail to detect this problem because entropy only measures diversity within a single input, not whether the model's reasoning actually changes across different inputs. They propose decomposing reasoning quality into two dimensions: within-input diversity (measured by entropy) and cross-input distinguishability (measured by mutual information), and develop mutual information proxies for real-time diagnosis during training. Across diverse tasks, their approach exposes template collapse as an invisible failure mode that existing metrics completely miss, enabling better detection and prevention of reasoning degradation in agentic systems.
Core Technical Contribution
The core innovation is the identification and formalization of template collapse as a distinct failure mode in RL-trained agents, fundamentally different from entropy collapse that prior work focused on. The authors introduce a two-dimensional decomposition of reasoning quality: entropy (within-input diversity) and mutual information (cross-input distinguishability), showing these measure orthogonal aspects of model behavior. They develop a family of mutual information proxies that can be computed efficiently during online training, enabling real-time detection without expensive offline analysis. This shifts the diagnosis paradigm from single-metric tracking (entropy alone) to a multi-faceted evaluation that catches failure modes invisible to existing diagnostics.
How It Works
The mechanism operates by first collecting reasoning trajectories from the multi-turn RL agent during training on diverse inputs. For each trajectory sample, the authors compute entropy over the probability distribution of tokens or reasoning steps within that single input, measuring output diversity. In parallel, they compute mutual information between input representations and reasoning output representations across the full batch or dataset, capturing whether different inputs produce meaningfully different reasoning patterns. The mutual information proxies (likely variations like variational bounds or empirical estimates) are designed to be differentiable and low-cost, enabling computation during the training loop itself. The diagnostic system compares these two metrics together: high entropy with low mutual information indicates template collapse (diverse-looking but input-agnostic outputs), while healthy reasoning shows both high entropy and high mutual information. This decomposition allows engineers to identify which dimension is failing and adjust training curriculum or reward signals accordingly.
Production Impact
In production agent systems, adopting this diagnostic framework prevents silent failures where agents appear to be reasoning properly (high entropy logs look good) but actually ignore user input, leading to repetitive or irrelevant outputs in multi-turn conversations. This is particularly critical for customer-facing applications where agents serve different users with different intents but generate the same canned responses. Engineers would integrate mutual information tracking into their RL training pipelines, adding a monitoring layer that flags template collapse early before deployment, reducing debugging time and production failures. The compute cost is moderate—mutual information proxies are designed for online efficiency—but requires batching strategies to compute cross-input statistics, potentially increasing training memory or adjusting batch sampling patterns. The practical benefit is immediate: a production pipeline can now detect a failure mode that would otherwise only show up through user complaints or manual monitoring of response patterns.
Limitations and When Not to Use This
The paper's scope is limited to RL-trained multi-turn agents and may not directly apply to supervised fine-tuning or inference-time sampling strategies, which operate under different dynamics. The mutual information proxies require sufficient batch diversity to estimate cross-input distinguishability accurately; in scenarios with small batch sizes or narrow input distributions, the signal may be noisy or uninformative. The paper assumes that input-responsiveness is always desirable, but there may be legitimate cases where agents benefit from partially template-driven behavior for consistency or safety reasons. Additionally, the abstract suggests the work is incomplete (cuts off mid-sentence), so the full set of results, benchmarks tested, and practical guidelines for practitioners are not visible in the provided text.
Research Context
This work builds on the established problem of training instability in RL-based LLM agents and extends beyond prior entropy-based diagnostics that dominated stability research. It relates to broader work on interpretability and failure mode detection in large language models, particularly research on understanding what language models actually learn versus what metrics suggest they've learned. The paper contributes to the emerging subfield of agent robustness, where understanding pathological behaviors (like template collapse) is essential for reliable agentic AI systems. It opens a research direction toward multi-metric diagnostic frameworks and suggests that simple aggregate metrics may systematically miss important failure modes, relevant for safety and reliability work across RL-trained systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::