InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 1 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Bohan Hou et al. |
| Year | 2026 |
| HF Upvotes | 1 |
| arXiv | 2605.07510 |
| Download | |
| Code | https://github.com/hbhalpha/InterLV-Search-Bench |
Abstract
Existing benchmarks for multimodal agentic search evaluate multimodal search and visual browsing, but visual evidence is either confined to the input or treated as an answer endpoint rather than part of an interleaved search trajectory. We introduce InterLV-Search, a benchmark for Interleaved Language-Vision Agentic Search, in which textual and visual evidence is repeatedly used to condition later search. It contains 2,061 examples across three levels: active visual evidence seeking, controlled offline interleaved multimodal search, and open-web interleaved multimodal search. Beyond existing benchmarks, it also includes multimodal multi-branch samples that involve comparison between multiple entities during the evidence search. We construct Level 1 and Level 2 with automated pipelines and Level 3 with a machine-led, human-supervised open-web pipeline. We further provide InterLV-Agent for standardized tool use, trajectory logging, and evaluation. Experiments on proprietary and open-source multimodal agents show that current systems remain far from solving interleaved multimodal search, with the best model below 50% overall accuracy, highlighting challenges in visual evidence seeking, search control, and multimodal evidence integration. We release the benchmark data and evaluation code at https://github.com/hbhalpha/InterLV-Search-Bench
Engineering Breakdown
Plain English
InterLV-Search is a benchmark with 2,061 test cases that evaluates AI agents on multimodal search tasks where text and visual evidence are repeatedly used to refine subsequent searches. Unlike existing benchmarks that treat vision as either static input or final output, this one models realistic search workflows where an agent uses images it finds to make better decisions about what to search for next, including scenarios where agents must compare multiple entities.
Key Engineering Insight
The core technical contribution is formalizing interleaved multimodal reasoning as a control problem: agents must decide when to gather visual evidence, what that evidence means, and how it constrains the next search action. This requires systems to maintain context across modalities and handle branching decision trees rather than linear pipelines.
Why It Matters for Engineers
Production search and browsing agents today either treat vision as decorative or use it only at the end of a search chain. This benchmark forces engineers to build systems that genuinely integrate visual understanding into the search loop itself—feeding images back as input to condition the next query. This is essential for building agents that can handle tasks like product comparison or research where humans naturally switch between reading text and examining images iteratively.
Research Context
Previous multimodal search benchmarks either tested single-turn vision understanding or evaluated final answer quality after visual lookup. InterLV-Search advances this by testing agents on multi-step reasoning where vision and language genuinely interleave. It enables evaluation of more realistic agent workflows and exposes whether current models actually learn to use visual feedback strategically rather than treating it as a one-way input.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::