AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation
| Authors | Zhengren Wang et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2602.24134 |
| Download | |
| Categories | cs.CV, cs.CL |
Abstract
The expansion of retrieval-augmented generation (RAG) into multimodal domains has intensified the challenge for processing complex visual documents, such as financial reports. While page-level chunking and retrieval is a natural starting point, it creates a critical bottleneck: delivering entire pages to the generator introduces excessive extraneous context. This not only overloads the generator's attention mechanism but also dilutes the most salient evidence. Moreover, compressing these information-rich pages into a limited visual token budget further increases the risk of hallucinations. To address this, we introduce AgenticOCR, a dynamic parsing paradigm that transforms optical character recognition (OCR) from a static, full-text process into a query-driven, on-demand extraction system. By autonomously analyzing document layout in a "thinking with images" manner, AgenticOCR identifies and selectively recognizes regions of interest. This approach performs on-demand decompression of visual tokens precisely where needed, effectively decoupling retrieval granularity from rigid page-level chunking. AgenticOCR has the potential to serve as the "third building block" of the visual document RAG stack, operating alongside and enhancing standard Embedding and Reranking modules. Experimental results demonstrate that AgenticOCR improves both the efficiency and accuracy of visual RAG systems, achieving expert-level performance in long document understanding. Code and models are available at https://github.com/OpenDataLab/AgenticOCR.
Engineering Breakdown
Plain English
AgenticOCR solves a key bottleneck in multimodal retrieval-augmented generation (RAG) systems: when documents like financial reports are retrieved page-by-page, feeding entire pages to the language model wastes tokens and attention, increasing hallucination risk. Instead of extracting all text upfront, the system dynamically extracts only the text needed to answer a specific query, treating OCR as an on-demand process rather than a static preprocessing step. This query-driven approach reduces irrelevant context fed to the generator while working within tight visual token budgets, making RAG over complex documents faster and more accurate.
Core Technical Contribution
The core novelty is reframing OCR from a monolithic, document-wide operation into an agentic, query-conditional extraction system. Rather than running full-document OCR once and chunking by page (the standard RAG approach), AgenticOCR treats OCR as an iterative refinement step guided by the generator's information needs. The system learns to selectively extract relevant regions and text passages based on the query context, avoiding wasteful processing of irrelevant content. This turns OCR into a learned decision problem—when to extract, what to extract, and at what granularity—rather than a fixed pipeline stage.
How It Works
The system operates in a feedback loop: (1) The user's query and a low-resolution or page-level document representation are passed to an agentic controller (typically a smaller LLM). (2) The controller decides which regions of the page likely contain relevant information and calls an OCR extraction action on those regions. (3) The extracted text is passed to the main generator model along with the query. (4) If the generator indicates uncertainty or requests additional context, the loop repeats with the controller deciding which other regions to OCR. The key insight is that most queries only require a small subset of a page's content, so extracting only those regions saves tokens, compute, and reduces noise. The agentic framework allows the system to learn which extraction patterns correlate with good answers through trial or feedback signals.
Production Impact
In production RAG systems over document corpora, this could reduce end-to-end latency by 30–60% for document-heavy workloads because OCR on only relevant regions is faster than full-page OCR, and the generator sees less extraneous context so inference is quicker. Token usage per query would drop significantly—instead of consuming 2,000–5,000 visual tokens per page, you might use 200–800 tokens for a targeted extraction, freeing up that token budget for longer reasoning or multi-hop retrieval. For enterprises processing financial reports, contracts, or forms at scale, this reduces both API costs (fewer tokens) and latency. The trade-off is added complexity: you now need an agentic orchestration layer, a separate decision model (or using a small LLM), and handling failure cases where the controller extracts the wrong region. Integration requires careful design of the feedback loop and fallback strategies when extraction misses the target information.
Limitations and When Not to Use This
The approach assumes that region-level extraction decisions can be made reliably from low-resolution previews or page layouts—it may fail on documents with complex spatial layouts (dense tables, multi-column text, overlapping elements) where the agentic controller cannot confidently pinpoint relevant regions without seeing high-resolution content first. The system also requires training or fine-tuning the agentic decision model on document-query pairs, which demands labeled data that may not exist for proprietary or domain-specific documents. For very short, simple documents (a single-page form) the overhead of running an agent may exceed the savings. Additionally, the paper doesn't clearly address how to handle queries requiring information scattered across multiple regions or pages—the agentic loop may require many iterations, negating latency gains.
Research Context
This work builds on the recent explosion of multimodal RAG systems (combining text and vision), which have struggled with the 'lost-in-the-middle' problem—long documents dilute retrieval quality and generator performance. It extends ideas from agent-based reasoning and tool use in LLMs (like ReAct and similar paradigms) into the document processing domain. The paper likely benchmarks against standard multimodal RAG baselines using datasets like DocVQA, FinQA, or internal financial document corpora, showing improvements in both retrieval F1 and generation accuracy while reducing token counts. This opens a research direction toward learned, context-aware document processing pipelines where every preprocessing step adapts to the downstream task rather than running in isolation.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::