Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing
| Authors | Anmol Gulati et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.06503 |
| Download | |
| Categories | cs.CL |
Abstract
Recent advances in multimodal Retrieval-Augmented Generation (RAG) enable Large Language Models (LLMs) to analyze enterprise spreadsheet workbooks containing millions of cells, cross-sheet dependencies, and embedded visual artifacts. However, state-of-the-art approaches exclude critical context through single-pass retrieval, lose data resolution through compression, and exceed LLM context windows through naive full-context injection, preventing reliable multi-step reasoning over complex enterprise workbooks. We introduce Beyond Rows to Reasoning (BRTR), a multimodal agentic framework for spreadsheet understanding that replaces single-pass retrieval with an iterative tool-calling loop, supporting end-to-end Excel workflows from complex analysis to structured editing. Supported by over 200 hours of expert human evaluation, BRTR achieves state-of-the-art performance across three frontier spreadsheet understanding benchmarks, surpassing prior methods by 25 percentage points on FRTR-Bench, 7 points on SpreadsheetLLM, and 32 points on FINCH. We evaluate five multimodal embedding models, identifying NVIDIA NeMo Retriever 1B as the top performer for mixed tabular and visual data, and vary nine LLMs. Ablation experiments confirm that the planner, retrieval, and iterative reasoning each contribute substantially, and cost analysis shows GPT-5.2 achieves the best efficiency-accuracy trade-off. Throughout all evaluations, BRTR maintains full auditability through explicit tool-call traces.
Engineering Breakdown
Plain English
This paper tackles a critical problem in enterprise AI: existing multimodal RAG systems fail when dealing with massive spreadsheets containing millions of cells, cross-sheet dependencies, and embedded images because they either lose information through compression, miss context through single-pass retrieval, or exceed LLM context windows through naive full injection. The authors introduce BRTR (Beyond Rows to Reasoning), an agentic framework that replaces one-shot retrieval with an iterative tool-calling loop, enabling LLMs to perform complex multi-step reasoning over enterprise Excel workbooks end-to-end. The system supports both analysis tasks (answering questions about spreadsheet data) and structured editing (making programmatic changes), backed by over 200 hours of development and evaluation. This enables reliable spreadsheet understanding at enterprise scale where prior approaches failed due to architectural limitations rather than raw model capability.
Core Technical Contribution
The core innovation is replacing single-pass dense retrieval with an iterative agentic loop that treats spreadsheet understanding as a sequential decision problem rather than a static information retrieval problem. Instead of upfront retrieval that compresses or loses information, BRTR lets the LLM decide what data to fetch at each reasoning step through tool calls, dynamically adapting its retrieval strategy based on intermediate results. This approach is architecturally novel because it explicitly models the multi-step nature of spreadsheet analysis—where understanding one cell dependency may reveal the need to inspect distant cells or different sheets. The framework also handles multimodal inputs (text, numbers, embedded charts/images) natively by treating vision tokens and structured data as first-class components in the tool-calling loop, rather than separate preprocessing steps.
How It Works
BRTR operates as an agentic loop where the LLM is given access to a set of tools for spreadsheet interaction: cell value lookup, formula inspection, cross-sheet navigation, visual artifact extraction, and structured range queries. Input is a natural language query or edit request about a spreadsheet workbook plus metadata describing sheet structure, cell locations, and embedded artifacts. The LLM generates tool-call sequences in a loop: it reasons about what data it needs, makes a tool call to fetch specific cells/regions/visuals, receives the result, and repeats until it has enough context to answer or perform the task. At each iteration, the system retrieves only the specific cells or ranges the LLM requests, avoiding both full-context injection and lossy compression. The output is either a natural language answer backed by provenance (which cells were consulted) or structured edit instructions (cell assignments, formula changes) that preserve spreadsheet integrity by respecting dependencies. Critically, the agentic approach works within typical LLM context windows (4K–8K tokens) even for workbooks with millions of cells because retrieval is sparse and lazy rather than upfront and exhaustive.
Production Impact
For teams building enterprise data agents, this pattern eliminates a major blocker: the inability to reliably query and modify real company spreadsheets at scale. Today, production systems either (1) pre-index and compress spreadsheets heavily, losing fidelity and making edit safety hard to verify, or (2) feed entire workbooks to the LLM, which fails on large files and produces expensive, slow inference. BRTR's iterative retrieval approach lets you productionize spreadsheet QA and automation without these trade-offs. Implementation would involve: wrapping your spreadsheet library (openpyxl, xlwings, pandas) as LLM tools, adding a loop manager to handle agent steps, and building a provenance/validation layer to ensure edits are safe and traceable. The latency cost is higher per query (multiple LLM calls instead of one), but you gain accuracy, safety, and the ability to handle workbooks that would previously timeout or exceed context limits. This is especially valuable for finance, operations, and HR teams where spreadsheets are the source of truth and manual analysis is expensive.
Limitations and When Not to Use This
The paper's abstract doesn't detail evaluation metrics, so it's unclear how well BRTR handles truly adversarial cases: spreadsheets with circular dependencies, implicit relationships across distant sheets, or ambiguous natural language queries that require exploring large search spaces. The iterative tool-calling approach adds latency per query (multiple LLM forward passes), which may be prohibitive for latency-sensitive applications—the paper doesn't report wall-clock time or cost comparisons to baselines. There's also an implicit assumption that the LLM can reliably decide what to retrieve next, which breaks down if the spreadsheet schema is poorly documented or if the query requires global reasoning over structure the LLM can't infer from tool results. Finally, the approach requires careful tool design and error handling (what if a formula reference points to a deleted cell? what if vision extraction fails on an embedded image?), adding production engineering burden that the paper likely underemphasizes.
Research Context
This work builds on recent advances in agentic LLMs and tool-calling (code interpreters, ReACT, language-model-based planning) and extends them to the structured data domain where spreadsheets have been historically underserved by NLP. It addresses a gap between general multimodal RAG systems (designed for unstructured text and images) and domain-specific spreadsheet tools (which handle only rigid query languages), positioning the agentic loop as a middle ground. The paper likely compares against strong baselines like GPT-4 with vision + naive full-context injection, dense retrieval + compression, and traditional spreadsheet formula builders. The research direction is promising because spreadsheet understanding is a real, high-value problem in enterprise AI, and the agentic pattern (iterative retrieval + reasoning) is increasingly recognized as superior to one-shot dense retrieval in complex domains like code search and scientific knowledge bases.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::