AgentSearchBench: A Benchmark for AI Agent Search in the Wild
| Authors | Bin Wu et al. |
| Year | 2026 |
| HF Upvotes | 0 |
| arXiv | 2604.22436 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at https://github.com/Bingo-W/AgentSearchBench.
Engineering Breakdown
Plain English
AgentSearchBench addresses a critical gap in AI agent ecosystems: how to identify the right agent for a task when agent capabilities are compositional and execution-dependent. The paper introduces a large-scale benchmark built from nearly 10,000 real-world agents across multiple providers, formalizing agent search as a retrieval and reranking problem. Unlike prior work that assumes well-specified functionalities or controlled candidate pools, this benchmark tackles realistic agent discovery scenarios where textual descriptions alone are insufficient. The work provides both executable and non-executable task queries, enabling more comprehensive evaluation of agent search methods.
Core Technical Contribution
The core innovation is the formalization of agent search as a two-stage retrieval and reranking problem that explicitly accounts for compositional agent capabilities and execution-dependent behavior. Most prior benchmarks assume static, well-documented tool functionalities; AgentSearchBench introduces the first large-scale dataset of real-world agents with realistic variability in capability specification and documentation quality. The benchmark distinguishes between executable task queries (where agent code can be run) and non-executable ones (only textual descriptions available), capturing both offline and online evaluation scenarios. This dual-mode evaluation framework enables assessment of agent search methods under conditions that reflect actual deployment challenges.
How It Works
AgentSearchBench operates as a two-stage pipeline: (1) dense retrieval stage where a query (task description) is matched against 10,000 candidate agents using embedding-based retrieval, and (2) reranking stage where top-k candidates are reordered using either cross-encoder models or execution results. For executable tasks, the system can actually run candidate agents and observe their outputs, enabling ground-truth evaluation of whether an agent succeeds. For non-executable tasks, the pipeline relies on textual similarity metrics and learned ranking models. The benchmark includes diverse agent types from multiple providers (APIs, code-based agents, multi-step workflows), each with varying documentation quality and capability specifications. Evaluation metrics measure both ranking quality (recall@k, NDCG) and practical success—whether retrieved agents can actually solve the given task when executed.
Production Impact
For teams building agent platforms or AI orchestration systems, this benchmark provides concrete evaluation methodology for agent discovery pipelines—a critical bottleneck when scaling from hundreds to thousands of agents. A production agent marketplace needs to solve agent search in milliseconds while maintaining high recall; AgentSearchBench clarifies trade-offs between dense retrieval speed and reranking accuracy. Implementation would involve: (1) deploying a vector database indexed on agent descriptions/capabilities, (2) running a two-stage pipeline with a fast retriever followed by a slower but more accurate reranker, and (3) optional online refinement via agent execution to validate recommendations. The benchmark's distinction between executable and non-executable task queries directly maps to offline evaluation (cheap, batch) versus online evaluation (expensive, real-time) in production systems. Cost impact: dense retrieval is nearly free once indexes are built, but execution-based reranking costs scale with agent call frequency; latency impact is typically 50-200ms for retrieval + 100-500ms for reranking depending on model choice.
Limitations and When Not to Use This
The benchmark is limited to agent discovery given pre-existing candidate pools; it does not address agent composition or dynamic agent creation, which are increasingly important in complex task scenarios. The paper does not fully solve the problem of agents with poorly documented capabilities or emergent behaviors discovered only at runtime—the evaluation assumes agent specifications are reasonably accurate. Scalability beyond 10,000 agents is not directly addressed; latency and ranking quality degradation at 1M+ agent scale remains open. The benchmark may overestimate real-world performance if agent documentation quality systematically differs from the corpus used, or if task distributions in production diverge from the benchmark's task set. Additionally, cross-provider generalization (e.g., training on one agent ecosystem and testing on another) is not thoroughly evaluated, limiting insights into transfer learning for agent search.
Research Context
AgentSearchBench builds on decades of information retrieval and ranking research (BM25, learning-to-rank, dense retrievers) but applies these to the novel domain of AI agent discovery. It extends prior work on tool-use benchmarks (ToolBench, APIBench) by handling the additional complexity of compositional agent capabilities and execution-dependent behavior. The work is situated in the emerging field of agent orchestration and marketplace platforms, alongside research on agent frameworks (LangChain, AutoGPT) and multi-agent systems. The benchmark likely catalyzes follow-up work on: agent capability summarization (how to write effective agent descriptions), execution-efficient ranking (reducing cost of online evaluation), and cross-provider agent compatibility (agents calling other agents). This positions agent search as a first-class research problem comparable to information retrieval, opening opportunities for specialized architectures and training methods.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::