VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning
| Authors | Yucheng Shen et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2604.09508 |
| Download | |
| Categories | cs.CV, cs.AI |
Abstract
Visual Retrieval-Augmented Generation (VRAG) empowers Vision-Language Models to retrieve and reason over visually rich documents. To tackle complex queries requiring multi-step reasoning, agentic VRAG systems interleave reasoning with iterative retrieval.. However, existing agentic VRAG faces two critical bottlenecks. (1) Visual Evidence Sparsity: key evidence is scattered across pages yet processed in isolation, hindering cross-page reasoning; moreover, fine-grained intra-image evidence often requires precise visual actions, whose misuse degrades retrieval quality; (2) Search Drift in Long Horizons: the accumulation of visual tokens across retrieved pages dilutes context and causes cognitive overload, leading agents to deviate from their search objective. To address these challenges, we propose VISOR (Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning), a unified single-agent framework. VISOR features a structured Evidence Space for progressive cross-page reasoning, coupled with a Visual Action Evaluation and Correction mechanism to manage visual actions. Additionally, we introduce a Dynamic Trajectory with Sliding Window and Intent Injection to mitigate search drift. They anchor the evidence space while discarding earlier raw interactions, preventing context from being overwhelmed by visual tokens. We train VISOR using a Group Relative Policy Optimization-based Reinforcement Learning (GRPO-based RL) pipeline with state masking and credit assignment tailored for dynamic context reconstruction. Extensive experiments on ViDoSeek, SlideVQA, and MMLongBench demonstrate that VISOR achieves state-of-the-art performance with superior efficiency for long-horizon visual reasoning tasks.
Engineering Breakdown
Plain English
This paper addresses a critical problem in agentic Vision-Language Models that retrieve and reason over multi-page documents: existing systems fail when evidence is scattered across pages or when the agent's search objective gets diluted by accumulated visual context over long retrieval sequences. The authors identify two specific bottlenecks—Visual Evidence Sparsity (key information scattered across isolated pages) and Search Drift in Long Horizons (context overload causing agents to lose track of their original goal)—and propose solutions to enable more effective multi-step reasoning over visually rich documents. The work targets production retrieval-augmented generation systems that need to handle complex document understanding tasks requiring cross-page reasoning without degrading performance.
Core Technical Contribution
The paper's core innovation is a framework for agentic VRAG that explicitly addresses multi-page evidence integration and long-horizon context management. Rather than processing retrieved pages in isolation, the system implements mechanisms to maintain coherent cross-page reasoning while preventing visual token accumulation from diluting the agent's search objective. The technical novelty lies in engineering both a solution for fine-grained visual action precision (to avoid retrieval quality degradation) and a context management strategy that keeps agents focused on their original query intent across multiple retrieval iterations. This represents a fundamental shift from treating each retrieval step independently to architecting the agent's memory and reasoning state to maintain coherence across long interaction sequences.
How It Works
The agentic VRAG system operates in an iterative loop: given a user query, the agent performs reasoning to decide what visual evidence to retrieve, executes precise visual actions to locate and extract relevant content from documents, integrates retrieved information with previously gathered evidence, and repeats until sufficient information accumulates to answer the query. The key technical components address the two bottlenecks through (1) a cross-page reasoning module that maintains a unified representation of evidence across multiple pages rather than processing pages in isolation, and (2) a context management strategy that separates the search objective from the accumulated visual tokens, preventing the agent from becoming cognitively overloaded. Fine-grained visual actions are guided by attention mechanisms or spatial localization to ensure retrieval precision, while the agent's state maintains an explicit representation of its current goal to prevent drift during long retrieval horizons. The system likely employs iterative refinement where each retrieval cycle updates both the evidence store and a goal-tracking mechanism.
Production Impact
In production systems, this approach directly improves handling of complex document-based queries that currently fail or degrade when evidence spans multiple pages—a common scenario in financial documents, legal contracts, academic papers, or technical specifications. Teams building document understanding pipelines would see immediate value in cross-page reasoning without building separate page-aggregation preprocessing steps, reducing pipeline complexity and latency. The long-horizon stability guarantees mean agents can be deployed with confidence on multi-step retrieval tasks (5-10+ iterations) without performance collapse, which is critical for real-world document analysis where answers rarely exist on a single page. Trade-offs include increased compute cost per query due to iterative retrieval and context management overhead, longer inference latency per query (likely 2-3x baseline VRAG), and greater memory footprint to maintain cross-page evidence representations. Integration requires careful calibration of the agent's stopping criteria and context window sizing to balance retrieval completeness against token budget constraints.
Limitations and When Not to Use This
The paper does not address computational scaling for extremely large document collections (100k+ pages), where retrieval latency itself becomes a bottleneck independent of the reasoning quality. The approach assumes visual evidence can be meaningfully located via precise actions, which may fail on documents with dense layouts, scanned PDFs with OCR errors, or layouts with ambiguous spatial relationships. The method likely requires careful tuning of the context management threshold—too aggressive and it discards useful information, too lenient and it reintroduces the original drift problem—without clear guidance on setting this parameter across different document domains. The paper does not evaluate robustness to adversarial document layouts or cases where the agent's objective is inherently ambiguous, nor does it address how cross-page reasoning performs when relevant evidence is fragmented across 10+ pages rather than 2-3.
Research Context
This work builds on the recent wave of retrieval-augmented generation (RAG) systems for language models, extending that framework to vision-language models that must reason over spatial and visual properties of documents. It directly addresses failure modes observed in prior agentic RAG systems (like ReAct and similar agent frameworks) when applied to multi-document reasoning, particularly the long-horizon context degradation problem identified in recent agent scaling studies. The paper contributes to the broader research direction of agent reliability and coherence, which is an active area given increasing deployment of LLM-based agents in production systems. By focusing specifically on visual documents rather than plain text, it opens up applications in enterprise document automation, where VLMs have previously struggled with multi-page reasoning tasks that humans perform routinely.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::