Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation
| Authors | Peiyang Liu et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2605.01284 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Iterative Retrieval-Augmented Generation (iRAG) has emerged as a powerful paradigm for answering complex multi-hop questions by progressively retrieving and reasoning over external documents. However, current systems predominantly operate on parsed text, which creates two critical bottlenecks: (1) Coarse-grained attribution, where users are burdened with manually locating evidence within lengthy documents based on vague text-level citations; and (2) Visual semantic loss, where the conversion of visually rich documents (e.g., slides, PDFs with charts) into text discards spatial logic and layout cues essential for reasoning. To bridge this gap, we present Chain of Evidence (CoE), a retriever-agnostic visual attribution framework that leverages Vision-Language Models to reason directly over screenshots of retrieved document candidates. CoE eliminates format-specific parsing and outputs precise bounding boxes, visualizing the complete reasoning chain within the retrieved candidate set. We evaluate CoE on two distinct benchmarks: Wiki-CoE, a large-scale dataset of structured web pages derived from 2WikiMultiHopQA, and SlideVQA, a challenging dataset of presentation slides featuring complex diagrams and free-form layouts. Experiments demonstrate that fine-tuned Qwen3-VL-8B-Instruct achieves robust performance, significantly outperforming text-based baselines in scenarios requiring visual layout understanding, while establishing a retriever-agnostic solution for pixel-level interpretable iRAG. Our code is available at https://github.com/PeiYangLiu/CoE.git.
Engineering Breakdown
Plain English
This paper introduces Chain of Evidence (CoE), a framework that fixes a major usability problem in retrieval-augmented generation systems: when models pull in documents to answer complex multi-hop questions, they cite text snippets that users then have to hunt through manually. The authors leverage Vision-Language Models to work directly with the visual layout and structure of documents (PDFs, slides, charts) instead of converting everything to plain text first, which loses crucial spatial information. The core innovation is making attribution fine-grained and visually grounded—pointing to exact locations in documents rather than vague text passages—while preserving the visual semantics that help reasoning over complex layouts.
Core Technical Contribution
The technical novelty is a retriever-agnostic visual attribution framework that integrates Vision-Language Models into the iterative RAG pipeline to ground citations at the spatial level rather than just the text level. Instead of converting visual documents to text and losing layout information, CoE reasons over the original visual representation, enabling the model to reference specific locations (e.g., coordinates in a chart or table cell) that users can immediately locate. This is a paradigm shift from coarse-grained text-level attribution to fine-grained visual attribution, and the approach is retriever-agnostic, meaning it can work with different document retrieval backends without modification.
How It Works
The pipeline works as follows: (1) documents are retrieved in their original visual format (PDF, image, etc.) rather than being converted to plain text; (2) a Vision-Language Model processes the visual representation alongside the query to identify both content relevance and spatial location of relevant information; (3) during the reasoning step, when the model generates an answer, it simultaneously outputs spatial coordinates or visual pointers (bounding boxes, page regions, etc.) that map back to the original document layout; (4) in multi-hop scenarios, the framework iteratively retrieves new documents based on reasoning outcomes from the previous step, maintaining visual grounding throughout; (5) the final output includes both the answer text and precise visual citations that users can click or navigate to directly. The key is that the Vision-Language Model acts as both a semantic reasoner and a spatial locator, eliminating the lossy text conversion step.
Production Impact
For teams building document-heavy question-answering systems (legal, financial, medical), this dramatically improves user experience: instead of telling a user 'the answer is in company_report_2024.pdf,' you can highlight the exact chart, table row, or paragraph. This reduces manual evidence verification time from minutes per query to seconds, especially valuable in regulatory or compliance scenarios where traceability is non-negotiable. The trade-off is increased compute: processing full-resolution PDFs through Vision-Language Models costs more than text-only processing, and latency will be higher per-retrieval step. Integration complexity is moderate—you need to store documents in visual format (not just text indices) and swap your citation mechanism, but the framework is retriever-agnostic so existing retrieval pipelines can be adapted. For organizations with millions of documents, the storage and inference cost per query will be 2-5x higher than text-only RAG, but the reduction in human verification work often justifies the cost.
Limitations and When Not to Use This
The framework assumes that documents are available in visual form (PDFs, images) and that spatial layout contains meaningful semantic information—this breaks down for scanned documents with OCR errors or when reasoning depends purely on text content independent of layout. The paper doesn't address how well this scales to extremely long documents or document collections with thousands of pages, where token limits in Vision-Language Models become a bottleneck; the computational cost of processing high-resolution imagery at scale may be prohibitive. There's no discussion of how performance degrades with low-quality scans, rotated text, or handwritten annotations, and it's unclear whether the visual attribution is robust to minor layout changes (reformatting, font changes, margin adjustments). The approach also doesn't solve the problem of reasoning across documents with incomparable visual structures—if you need to synthesize information from 10 documents with different layouts, aligning spatial reasoning becomes difficult.
Research Context
This work builds on the iterative RAG paradigm (which chains retrieval and reasoning to handle multi-hop questions) and extends it with visual grounding, addressing a gap that prior work like Dense Passage Retrieval and RAG-based systems didn't solve: fine-grained, spatial attribution. It fits into the broader trend of Vision-Language Models (GPT-4V, Claude 3.5 Vision) being integrated into information retrieval pipelines, moving away from text-only assumptions. The research opens up new evaluation benchmarks around visual citation accuracy and user-study metrics on citation utility, and it likely influences future work on multimodal retrieval, where visual and textual modalities are processed jointly rather than sequentially.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::