Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 81 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Zhuofeng Li et al. |
| Year | 2026 |
| HF Upvotes | 81 |
| arXiv | 2605.05242 |
| Download | |
| Code | https://github.com/DCI-Agent/DCI-Agent-Lite |
Abstract
Modern retrieval systems, whether lexical or semantic, expose a corpus through a fixed similarity interface that compresses access into a single top-k retrieval step before reasoning. This abstraction is efficient, but for agentic search, it becomes a bottleneck: exact lexical constraints, sparse clue conjunctions, local context checks, and multi-step hypothesis refinement are difficult to implement by calling a conventional off-the-shelf retriever, and evidence filtered out early cannot be recovered by stronger downstream reasoning. Agentic tasks further exacerbate this limitation because they require agents to orchestrate multiple steps, including discovering intermediate entities, combining weak clues, and revising the plan after observing partial evidence. To tackle the limitation, we study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools (e.g., grep, file reads, shell commands, lightweight scripts), without any embedding model, vector index, or retrieval API. This approach requires no offline indexing and adapts naturally to evolving local corpora. Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines on several BRIGHT and BEIR datasets, and attains strong accuracy on BrowseComp-Plus and multi-hop QA without relying on any conventional semantic retriever. Our results indicate that as language agents become stronger, retrieval quality depends not only on reasoning ability but also on the resolution of the interface through which the model interacts with the corpus, with which DCI opens a broader interface-design space for agentic search.
Engineering Breakdown
Plain English
This paper identifies a fundamental bottleneck in how modern retrieval systems work for agentic AI: they force all corpus access through a single top-k similarity lookup, which discards evidence permanently and prevents agents from performing complex multi-step reasoning like hypothesis refinement, constraint checking, and clue combination. The authors argue that agentic search requires direct, flexible corpus interaction rather than the fixed abstraction that makes traditional retrievers efficient.
Key Engineering Insight
Retrieval systems need to expose the corpus as an interactive surface that agents can query iteratively with different constraints and logic, rather than compressing all access into a single similarity-based top-k call. This shift from 'retriever as black box' to 'retriever as tool' changes how you architect the retrieval layer.
Why It Matters for Engineers
If you're building multi-step AI agents that search for evidence—like fact-checking systems, research assistants, or reasoning engines—you've hit this problem: early retrieval decisions are irreversible and agents can't recover evidence needed for later reasoning steps. This forces engineers to either over-retrieve (expensive) or re-architect their entire pipeline. This paper directly addresses the engineering tension between efficiency and the dynamic reasoning needs of real agent deployments.
Research Context
Previous work treated retrieval as a fixed preprocessing step: semantic search solved ranking, lexical search solved exact matches, but neither handled agentic workflows that need to explore hypotheses, combine sparse signals, and backtrack. This paper shifts the paradigm from 'optimizing a single retrieval call' to 'designing retrieval as an interactive agent interface,' enabling closer coupling between reasoning and corpus exploration.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::