A^2TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 11 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Dingwei Chen et al. |
| Year | 2026 |
| HF Upvotes | 11 |
| arXiv | 2605.06200 |
| Download | |
| Code | https://github.com/CuSO4-Chen/A-TGPO |
Abstract
Reinforcement learning for agentic large language models (LLMs) typically relies on a sparse, trajectory-level outcome reward, making it difficult to evaluate the contribution of individual tool-calls within multi-turn interactions. Existing approaches to such process credit assignment either depend on separate external process reward models that introduce additional consumption, or tree-based structural rollout that merely redistributes the outcome signal while constraining trajectory diversity. A promising alternative leverages the per-turn change in the policy's predicted probability of the ground-truth, termed Information Gain (IG), as an intrinsic process signal without an external evaluator. However, prior work on leveraging IG signals within the RL training loop faces three systematic challenges: normalizing across turns that face heterogeneous positional contexts can distort the relative standing of individual turns, accumulating a variable number of terms causes advantage magnitudes to drift with trajectory depth, and a fixed clipping range governs policy updates identically for turns with vastly different IG signals. In this paper, we propose A^2TGPO (Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping), which retains IG as the intrinsic signal but re-designs how it is normalized, accumulated, and consumed: (i) turn-group normalization: normalizes IG within each (prompt, turn-index) group so that each turn is compared only against peers at the same interaction depth; (ii) variance-rescaled discounted accumulation: divides cumulative normalized IG by square root of accumulated terms to keep advantage magnitudes comparable across turn positions; and (iii) adaptive turn-level clipping: modulates each turn's clipping range based on its normalized IG, widening the update region for informative turns and narrowing it for uninformative ones.
Engineering Breakdown
Plain English
This paper proposes A²TGPO, a reinforcement learning method for agentic LLMs that solves the credit assignment problem in multi-turn interactions without needing external reward models. Instead of relying on sparse final outcomes or expensive process rewards, it uses Information Gain (IG)—the per-turn change in the model's predicted probability of the correct answer—as a free, intrinsic signal to train better tool-use policies.
Key Engineering Insight
Using Information Gain as an intrinsic process signal eliminates the need for external reward model evaluators while maintaining trajectory diversity. The key innovation is adaptive turn-level clipping that prevents pathological policy behavior when IG signals are noisy or sparse, making the approach practically stable.
Why It Matters for Engineers
Production agentic systems struggle with the cost and latency of external process reward models for each tool call, yet sparse outcome rewards make training inefficient. This method reduces inference overhead by computing IG directly from the policy's own outputs, cutting the computational cost of RL fine-tuning while improving per-action credit assignment in long multi-step interactions.
Research Context
Prior work tried to solve credit assignment in multi-turn RL either by training separate reward models (expensive) or using tree-search rollouts (limited diversity). IG-based approaches showed promise but were unstable in practice. This paper advances the field by introducing adaptive clipping mechanisms that make IG signals reliable enough for production RL loops, enabling cheaper and faster training of tool-using agents.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::