T^2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-04 with 8 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Haixin Wang et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2605.02178 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent progress in multi-turn reinforcement learning (RL) has significantly improved reasoning LLMs' performances on complex interactive tasks. Despite advances in stabilization techniques such as fine-grained credit assignment and trajectory filtering, instability remains pervasive and often leads to training collapse. We argue that this instability stems from inefficient exploration in multi-turn settings, where policies continue to generate low-information actions that neither reduce uncertainty nor advance task progress. To address this issue, we propose Token- and Turn-level Policy Optimization (T^2PO), an uncertainty-aware framework that explicitly controls exploration at fine-grained levels. At the token level, T^2PO monitors uncertainty dynamics and triggers a thinking intervention once the marginal uncertainty change falls below a threshold. At the turn level, T^2PO identifies interactions with negligible exploration progress and dynamically resamples such turns to avoid wasted rollouts. We evaluate T^2PO in diverse environments, including WebShop, ALFWorld, and Search QA, demonstrating substantial gains in training stability and performance improvements with better exploration efficiency. Code is available at: https://github.com/WillDreamer/T2PO.
Engineering Breakdown
Plain English
This paper addresses training instability in multi-turn reinforcement learning for reasoning LLMs by proposing T^2PO, a framework that controls exploration at both token and turn levels using uncertainty monitoring. The core finding is that instability stems from policies generating low-information actions during exploration, and by selectively triggering interventions when uncertainty is high, the system achieves more stable training without collapse.
Key Engineering Insight
The insight is that exploration efficiency matters more than just exploration quantity in multi-turn RL—the system should actively intervene when the model is uncertain and not making progress, rather than letting it freely explore. This shifts the paradigm from passive trajectory filtering to active, uncertainty-driven intervention.
Why It Matters for Engineers
Training instability in agentic RL systems currently forces engineers to use expensive workarounds like frequent checkpoint rollbacks, conservative learning rates, or trajectory filtering heuristics. A principled uncertainty-guided approach could reduce training time, improve convergence reliability, and lower the computational cost of scaling reasoning models to handle longer, more complex interactive tasks in production.
Research Context
Prior work stabilized multi-turn RL through credit assignment and trajectory filtering, but these are post-hoc patches that don't prevent low-value exploration. T^2PO advances the field by making exploration itself the control lever—monitoring uncertainty in real-time and intervening at token/turn granularity. This enables the next generation of more reliable agentic systems that can handle longer chains of reasoning without training collapse.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::