AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
| Authors | Lei Xiong et al. |
| Year | 2026 |
| HF Upvotes | 28 |
| arXiv | 2604.25256 |
| Download | |
| Code | https://github.com/CherYou/AutoResearchBench |
Abstract
Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents' capability in driving this process, we present AutoResearchBench, a dedicated benchmark for autonomous scientific literature discovery. AutoResearchBench consists of two complementary task types: (1) Deep Research, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) Wide Research, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, AutoResearchBench is distinguished along three dimensions: it is research-oriented, calling for in-depth comprehension of scientific concepts; literature-focused, demanding fine-grained utilization of detailed information; and open-ended, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make AutoResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%. We publicly release the dataset and evaluation pipeline to facilitate future research in this direction. We publicly release the dataset, evaluation pipeline, and code at https://github.com/CherYou/AutoResearchBench.
Engineering Breakdown
Plain English
This paper introduces AutoResearchBench, a benchmark designed to evaluate how well AI agents can autonomously discover scientific literature. The benchmark includes two complementary task types: Deep Research, where agents must locate a specific target paper through multi-step iterative search, and Wide Research, where agents must comprehensively collect papers matching specific criteria. This addresses a critical capability gap in autonomous scientific research systems, where finding and synthesizing relevant literature is foundational to hypothesis formation and validation. The benchmark fills a gap in existing agentic web evaluation tools by specifically targeting the literature discovery process, which is distinct from general web search or information retrieval.
Core Technical Contribution
The core contribution is the design and release of a dual-task benchmark that operationalizes scientific literature discovery as an agentic capability. Unlike prior work that treats search as a simple retrieval problem, AutoResearchBench models discovery as a sequential decision-making process where agents must iteratively refine queries, evaluate relevance, and track progress toward either a specific paper (Deep Research) or a comprehensive collection (Wide Research). The benchmark enables systematic evaluation of how well agents perform reasoning over scientific knowledge spaces, including handling citation networks, author relationships, and domain-specific terminology. This is the first dedicated benchmark for this specific problem in the agentic AI literature.
How It Works
For Deep Research tasks, an agent is given a target paper description and must progressively discover it through multi-step search interactions. The agent formulates initial queries based on the description, examines search results, extracts new information (authors, citations, terminology), and refines subsequent queries based on relevance feedback. This creates a state-space exploration problem where each action (a search query) yields information that constrains the next action. For Wide Research tasks, the agent receives criteria (e.g., papers on a specific topic from a date range with specific keywords) and must systematically collect all matching papers, requiring exhaustive exploration rather than targeted search. The benchmark likely provides API access to a scientific literature database and evaluates agents on metrics like coverage (papers found vs. total available), precision (false positives), and efficiency (steps/queries needed). Both task types test different aspects of agent reasoning: Deep Research emphasizes hypothesis refinement and constraint satisfaction, while Wide Research emphasizes systematic enumeration and completeness.
Production Impact
For teams building autonomous research systems or AI-assisted literature review tools, this benchmark provides concrete evaluation methodology and performance baselines. It enables developers to measure whether their agents can scale literature discovery beyond simple keyword matching—critical for scientific integrity where missing a key paper can invalidate claims. In production pipelines, adopting this evaluation approach means: (1) testing agents on multi-step discovery tasks rather than single-query retrieval, (2) measuring both precision and recall separately, and (3) instrumenting agent behavior to understand failure modes (e.g., query reformulation errors). The benchmark likely incurs computational cost through repeated API calls to literature databases; production systems would need to implement query caching and rate limiting. Integration complexity is moderate—teams need to connect their agents to scientific literature APIs (PubMed, arXiv, Semantic Scholar) and implement the benchmark's scoring metrics.
Limitations and When Not to Use This
The paper's scope appears limited to scientific literature discovery and does not address downstream tasks like reading comprehension, synthesis, or claim verification—agents may find papers but fail to extract or integrate their findings. The benchmark assumes access to structured scientific databases; performance may degrade significantly with noisy or incomplete metadata. The evaluation likely assumes English-language literature and standard scientific domains, so applicability to emerging fields or multilingual research is unclear. The paper does not discuss how benchmark difficulty scales with domain expertise required; a domain-naive agent might succeed on well-known papers but fail on niche literature, making it unclear whether high scores reflect true research capability or domain knowledge shortcuts.
Research Context
This work extends the emerging field of agentic AI benchmarking, building on frameworks like WebShop, ToolBench, and other web navigation/interaction benchmarks. It specifically addresses a gap identified in autonomous science literature where agents need structured evaluation on knowledge-intensive tasks. The benchmark contributes to the broader research direction of evaluating AI agents on realistic multi-step reasoning tasks rather than single-turn language understanding. It connects to work on agent planning, information retrieval under uncertainty, and knowledge graph exploration, positioning literature discovery as a key capability for end-to-end autonomous science systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::