Learning to Retrieve from Agent Trajectories
| Authors | Yuqi Zhou et al. |
| Year | 2026 |
| HF Upvotes | 67 |
| arXiv | 2604.04949 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Information retrieval (IR) systems have traditionally been designed and trained for human users, with learning-to-rank methods relying heavily on large-scale human interaction logs such as clicks and dwell time. With the rapid emergence of large language model (LLM) powered search agents, however, retrieval is increasingly consumed by agents rather than human beings, and is embedded as a core component within multi-turn reasoning and action loops. In this setting, retrieval models trained under human-centric assumptions exhibit a fundamental mismatch with the way agents issue queries and consume results. In this work, we argue that retrieval models for agentic search should be trained directly from agent interaction data. We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions. Through a systematic analysis of search agent trajectories, we identify key behavioral signals that reveal document utility, including browsing actions, unbrowsed rejections, and post-browse reasoning traces. Guided by these insights, we propose LRAT, a simple yet effective framework that mines high-quality retrieval supervision from agent trajectories and incorporates relevance intensity through weighted optimization. Extensive experiments on both in-domain and out-of-domain deep research benchmarks demonstrate that retrievers trained with LRAT consistently improve evidence recall, end-to-end task success, and execution efficiency across diverse agent architectures and scales. Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
Engineering Breakdown
Plain English
This paper identifies a fundamental mismatch in how retrieval systems are built: traditional IR systems are trained on human interaction data (clicks, dwell time), but modern LLM-powered search agents consume retrieval results differently and issue queries in different patterns. The authors propose training retrieval models directly from agent interaction trajectories instead of human logs, creating a retrieval system optimized for agentic consumption rather than human users. This shift from human-centric to agent-centric retrieval training represents a necessary evolution as agents become the primary consumers of search results in multi-turn reasoning loops.
Core Technical Contribution
The core novelty is reframing the retrieval training paradigm from human interaction signals to agent trajectory data. Rather than using traditional learning-to-rank approaches that rely on clicks and dwell time, the authors extract relevance signals from how agents actually interact with retrieved documents within their reasoning and action loops. The key insight is that agents have fundamentally different information needs and consumption patterns compared to humans—they may chain multiple retrieval calls, reason over partial results, or reformulate queries based on intermediate findings—and these patterns should directly inform how retrieval models are trained. This introduces a new task framing: optimizing retrieval specifically for downstream agent performance rather than optimizing for human click-through or engagement metrics.
How It Works
The approach collects agent trajectories—sequences of queries, retrieved documents, and agent actions/reasoning steps—from LLM-powered agents executing multi-turn tasks. These trajectories provide implicit relevance signals: documents that led to successful agent reasoning or task completion are treated as relevant, while documents that caused reasoning failures or dead ends are treated as irrelevant. The system then trains a retrieval model using these agent-derived labels, likely through a learning-to-rank or contrastive learning framework that optimizes for documents that facilitate agent progress. The key difference from human-centric training is the feedback source and optimization target: instead of human clicks indicating interest, the training signal comes from whether a document helped the agent successfully continue its reasoning chain toward task completion. This requires instrumenting agent systems to log full trajectories (query, retrieved documents, agent reasoning, final outcome) and using outcome signals or intermediate success metrics as supervision.
Production Impact
Adopting this approach would require significant architectural changes: you'd need to instrument your agent systems to collect full trajectory data, implement offline or online learning pipelines that train on agent interaction data rather than human logs, and version your retrieval models differently. For teams running production agentic search systems, this could dramatically improve agent success rates by removing the misalignment where retrieval optimizes for human satisfaction rather than agent task completion—potentially reducing hallucinations caused by irrelevant retrieved context or improving multi-turn reasoning quality. The trade-off is substantial: you need to continuously collect agent trajectory data (which requires deployed agents), build new data pipelines and labeling infrastructure, and maintain multiple retrieval models (one for humans, one for agents, or a hybrid). Compute costs could increase due to the need for larger training datasets and more frequent retraining cycles, and the cold-start problem becomes harder—new agent types or tasks may require retraining rather than reusing existing human-trained models.
Limitations and When Not to Use This
The paper assumes you have access to large volumes of agent interaction data, which is a significant barrier for many organizations just beginning to deploy agents—the approach may not generalize well to new agent architectures or tasks without retraining. It also doesn't address the challenge of extracting reliable relevance signals from trajectories; noisy or biased trajectories (e.g., from agents making systematic reasoning errors) could degrade the training data. The paper likely doesn't deeply explore how to handle distribution shift when agent capabilities improve or when the same retrieval model needs to serve multiple different agent types simultaneously. Additionally, there's an open question about whether agent-centric retrieval would hurt performance in hybrid scenarios where the same search system serves both human users and agents, or whether separate retrieval models would be needed.
Research Context
This work sits at the intersection of information retrieval and agentic AI, building on the long history of learning-to-rank methods while addressing the emerging challenge of retrieval-in-the-loop for LLM agents. It builds on prior work in retrieval augmentation (RAG) and agent frameworks like ReAct, but shifts the focus from 'how to retrieve better for humans' to 'how to retrieve better for agents.' The paper likely benchmarks on existing agent reasoning datasets or introduces new evaluation metrics centered on agent task completion rather than traditional IR metrics like NDCG or MRR. This opens a research direction that challenges the assumption that human-optimized IR systems are suitable for agent consumption, suggesting future work in co-optimization (training for both humans and agents), domain-specific agent retrieval models, and new evaluation frameworks tailored to agentic workflows.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::