E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning
| Authors | Weiyang Guo et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2604.09455 |
| Download | |
| Categories | cs.AI |
Abstract
While Large Language Models (LLMs) have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert "anchors" and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.
Engineering Breakdown
Plain English
E3-TIR addresses a critical bottleneck in training tool-using language model agents: existing approaches either explore inefficiently from scratch (Zero-RL) or require massive labeled datasets with expensive supervised fine-tuning followed by reinforcement learning (SFT-then-RL). The paper proposes a warm-up training paradigm that strategically blends three types of experience—expert demonstrations as anchors, expert-guided exploration around those anchors, and self-directed exploration—combined with a mixture-of-policies optimization mechanism. This hybrid approach mitigates mode collapse (where the agent gets stuck in narrow solution patterns) and data inefficiency, enabling agents to learn tool use more effectively in early training stages.
Core Technical Contribution
The core innovation is formulating agent training as dynamic integration of three distinct experience sources rather than relying on a single paradigm. Instead of choosing between pure exploration (Zero-RL) or pure imitation-then-RL, E3-TIR creates a curriculum where expert demonstrations serve as 'anchors'—reference points for what good trajectories look like—and the agent explores diverse variations around those anchors. The mix policy optimization mechanism learns to weight contributions from each experience type during training, allowing the system to smoothly transition from expert guidance toward self-exploration as the agent becomes competent. This is fundamentally different from prior work because it treats expert data not as the end goal (as in SFT) but as scaffolding for exploration.
How It Works
The training pipeline operates in phases: first, expert trajectories are decomposed into 'Expert Prefixes' (states where expert performance is strongest) that serve as anchors for exploration. Second, around each anchor, the agent performs 'Expert Guided' exploration—sampling action trajectories that follow expert-like patterns but with controlled variations to explore the local solution space. Third, the agent independently explores ('Self-Exploration') to discover patterns the expert demonstrations missed. At each training step, a learned mixture-of-policies network assigns soft weights to trajectories from each source (w_expert, w_guided, w_self summing to 1), allowing the training signal to dynamically shift emphasis. The RL objective optimizes cumulative reward weighted by these learned policies, while the policy network itself updates to maximize performance across all three experience sources. This creates a curriculum effect where early training leverages expert signal for stability, then gradually shifts toward self-exploration as the base policy matures.
Production Impact
For teams deploying tool-using agents (code generation, API interaction, information retrieval), E3-TIR reduces both the annotation burden and training instability significantly. Instead of collecting thousands of high-quality demonstrations or running full RL from random initialization (which is computationally expensive and often fails), you'd run a 3-5K step warm-up phase with modest expert data, then transition to standard RL or deployment. This cuts wall-clock training time and stabilizes convergence, translating to faster iteration cycles and lower infrastructure costs. The main trade-off is added engineering complexity: you need to implement the branching exploration mechanism, the mixture-of-policies network, and careful hyperparameter tuning for the experience weighting schedule. For systems already using expert demonstrations, the overhead is moderate—you're adding structured exploration around existing data rather than collecting entirely new resources.
Limitations and When Not to Use This
The paper assumes high-quality expert trajectories are available to initialize the anchor set, which may not hold for novel domains or safety-critical tasks where expert data is scarce or expensive. The approach's effectiveness depends heavily on the quality of the expert prefixes and the radius of guided exploration; if anchors are poorly chosen or exploration bounds are too tight, the agent may fail to discover critical solution patterns. The mixture-of-policies mechanism adds hyperparameters (weighting schedule, exploration variance, curriculum pacing) that likely require domain-specific tuning, reducing the 'plug-and-play' appeal. The paper does not address how this scales to very long-horizon tasks (100+ steps) or tasks with sparse reward signals, which remain challenging for most RL-based approaches. Follow-up work should explore automatic anchor selection, adaptive exploration bounds, and application to domains where expert demonstrations are fundamentally unavailable.
Research Context
This work sits at the intersection of imitation learning and reinforcement learning for agents, building on foundational research in curriculum learning and mixing multiple data sources (e.g., DAgger, GAIL). It directly addresses known failure modes of prior tool-use training paradigms: Zero-RL suffers from catastrophic forgetting and mode collapse without guidance, while SFT-then-RL hits plateaus because supervised data cannot capture the full reward landscape. The contribution connects to recent work on mixture-of-experts and dynamic weighting in multi-task learning, adapting those ideas to the agent training setting. E3-TIR opens a research direction toward 'warm-start RL' for agents, where the goal is not to maximize final performance but to stabilize early training—a problem increasingly relevant as LLM-based agents move from research to production systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::