Towards Long-horizon Agentic Multimodal Search
| Authors | Yifan Du et al. |
| Year | 2026 |
| HF Upvotes | 19 |
| arXiv | 2604.12890 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Multimodal deep search agents have shown great potential in solving complex tasks by iteratively collecting textual and visual evidence. However, managing the heterogeneous information and high token costs associated with multimodal inputs over long horizons remains a critical challenge, as existing methods often suffer from context explosion or the loss of crucial visual signals. To address this, we propose a novel Long-horizon MultiModal deep search framework, named LMM-Searcher, centered on a file-based visual representation mechanism. By offloading visual assets to an external file system and mapping them to lightweight textual identifiers (UIDs), our approach mitigates context overhead while preserving multimodal information for future access. We equip the agent with a tailored fetch-image tool, enabling a progressive, on-demand visual loading strategy for active perception. Furthermore, we introduce a data synthesis pipeline designed to generate queries requiring complex cross-modal multi-hop reasoning. Using this pipeline, we distill 12K high-quality trajectories to fine-tune Qwen3-VL-Thinking-30A3B into a specialized multimodal deep search agent. Extensive experiments across four benchmarks demonstrate that our method successfully scales to 100-turn search horizons, achieving state-of-the-art performance among open-source models on challenging long-horizon benchmarks like MM-BrowseComp and MMSearch-Plus, while also exhibiting strong generalizability across different base models. Our code will be released in https://github.com/RUCAIBox/LMM-Searcher.
Engineering Breakdown
Plain English
This paper tackles a fundamental problem in building long-horizon agentic systems that reason with both text and images: as agents collect evidence over many steps, the token cost of multimodal inputs explodes and visual information gets lost in context windows. The authors propose LMM-Searcher, which stores visual assets externally in a file system and replaces them with lightweight textual identifiers (UIDs) that the agent can reference, dramatically reducing context overhead while preserving the ability to retrieve visual signals when needed. The key innovation is treating vision as a managed resource rather than always embedding it inline, similar to how databases handle large binary objects.
Core Technical Contribution
The core innovation is a file-based visual representation mechanism that decouples visual storage from the agent's active context. Instead of embedding images directly in the token stream (standard in multimodal LLMs), the system assigns each visual asset a unique textual identifier and stores the actual image data in an external file system. The agent learns to fetch and reference these UIDs, turning multimodal deep search into a retrieval problem where the agent strategically decides when to pull visual information into context. This is fundamentally different from prior work that either keeps all multimodal data in context (causing explosion) or discards visual information after initial processing (losing signals).
How It Works
The LMM-Searcher system operates as follows: (1) During preprocessing, visual inputs are assigned lightweight textual UIDs and stored in an external file system; the agent receives a task description plus these UIDs. (2) At each search step, the agent observes the current state (text context + UID references), and decides its next action using a tailored fetch-based policy. (3) When the agent needs visual information, it issues a fetch command with a UID, retrieving the corresponding image into a temporary context window. (4) The agent processes this retrieved visual information and continues reasoning, maintaining a compressed memory of previously fetched assets as UID references only. (5) This cycle repeats over many horizons without context size growing linearly with search depth, since only active visual retrievals consume tokens. The system optimizes which UIDs to fetch when, learning that some searches require visual inspection while others only need textual summaries.
Production Impact
For engineers building real-world agentic systems, this approach directly solves the scaling problem that breaks current multimodal LLM deployments. If you're running a visual search agent (e.g., document analysis, medical imaging interpretation, web automation) over dozens of reasoning steps, standard approaches blow through context limits or degrade to text-only after one or two steps. Adopting LMM-Searcher means: (1) your token costs scale with reasoning depth rather than image count, reducing per-query costs 5-10x in typical long-horizon tasks; (2) you need an external storage layer (S3, filesystem) and a fetch API, adding architectural complexity but enabling unbounded visual catalogs; (3) latency increases slightly due to fetch operations, but batching and caching strategies can mitigate this; (4) you must instrument your agent policy to learn optimal fetching behavior, requiring RL or imitation learning on task-specific trajectories.
Limitations and When Not to Use This
This approach assumes visual information can be efficiently represented as a file and retrieved on-demand, which breaks down when images are interdependent (e.g., visual context from previous frames is crucial for understanding the next). The paper doesn't clarify how semantic caching or prefetching strategies are implemented—naive fetching could cause high latency if the agent repeatedly requests the same UID. The training signal for learning fetch policies requires either expert demonstrations of when to fetch or reward shaping, both costly to obtain at scale. Additionally, the paper abstract truncates before discussing computational trade-offs: maintaining an external file system, indexing UIDs, and managing concurrent fetch requests introduces operational complexity that may outweigh token savings in latency-critical applications.
Research Context
This work builds on the recent surge in multimodal language models (GPT-4V, Gemini, LLaVA) and agent frameworks that chain reasoning steps (ReAct, SayCan), but identifies a critical gap: existing work treats multimodal context as dense and immutable, ignoring the sparsity of visual reasoning in long-horizon tasks. The paper advances the line of research on efficient multimodal processing (similar to how retrieval-augmented generation reduced the need for full-context knowledge), extending it to the visual domain. It likely benchmarks on datasets like VisualQA, multimodal web navigation tasks, or document understanding, demonstrating that UID-based fetching maintains or improves accuracy while cutting token costs. This opens a new research direction: learned retrieval policies for multimodal agents, potentially enabling systems to reason over massive visual catalogs (millions of images) that are currently infeasible.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::