Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
| Authors | Martin Asenov et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.04238 |
| Download | |
| Categories | cs.CL |
Abstract
Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.
Engineering Breakdown
Plain English
This paper challenges the assumption that modern end-to-end multimodal retrievers are fundamentally better than classical BM25 for retrieval-augmented generation (RAG) systems. The authors systematically vary how documents are transcribed and preprocessed while keeping the retrieval mechanism constant, and find that BM25 can close large performance gaps on multilingual and visually-complex document benchmarks. The key finding is that document representation quality—not the retrieval algorithm itself—drives most benchmark improvements. This suggests that practitioners may be overestimating the value of complex neural retrievers and underestimating the impact of better preprocessing pipelines.
Core Technical Contribution
The paper's core contribution is a methodological framework for decomposing RAG system improvements into representation quality versus retrieval mechanism components. Rather than proposing a new algorithm, the authors provide evidence through controlled experiments that careful transcription and preprocessing of documents can recover performance gaps that are typically attributed to neural retrievers. They challenge the implicit assumption in the field that end-to-end learning of document embeddings is necessary for strong multilingual and visual document retrieval. This decomposed evaluation approach shifts focus from architectural complexity to data quality and representation engineering.
How It Works
The experimental approach isolates variables by implementing a fixed retrieval mechanism (BM25) and systematically varying upstream processing stages. First, raw documents with visual layouts and multilingual text are fed into different transcription pipelines (OCR quality, text extraction methods, layout preservation strategies). Second, preprocessing steps modify the text representation (language-specific tokenization, normalization, structural annotation). Third, these representations are indexed and retrieved using the same BM25 ranking function, measuring performance on established benchmarks. By holding the retrieval algorithm constant and measuring performance deltas across preprocessing variations, the authors quantify the contribution of representation quality independent of algorithmic sophistication. This isolates which performance gains come from better document understanding versus better retrieval mechanisms.
Production Impact
For teams building RAG systems, this paper suggests a more pragmatic path than investing in expensive multimodal embedding models: focus engineering effort on document preprocessing, transcription quality, and representation engineering. A production RAG pipeline could potentially achieve state-of-the-art retrieval performance with BM25 plus a well-engineered document preprocessing stage, rather than training or fine-tuning large neural retrievers. This reduces infrastructure costs (no need for GPU-based embedding inference at scale), simplifies the tech stack (BM25 is mature, well-understood, and fast), and improves debuggability and interpretability of retrieval failures. However, the trade-off is that this approach requires more domain expertise in preprocessing and OCR pipeline tuning, and may not benefit from newer research in dense retrieval as quickly as end-to-end models do.
Limitations and When Not to Use This
The paper's findings are scoped to the specific benchmarks and document types evaluated; results may not generalize to domains with different visual complexity, language distributions, or query-document matching characteristics. The work assumes that preprocessing improvements are the primary lever, but doesn't fully address scenarios where documents are dynamically updated or streaming (where reprocessing is expensive) or where the query-document semantic gap is fundamentally large. The paper doesn't provide guidance on how to systematically discover which preprocessing methods matter most for a new domain or document type, so applying these insights requires substantial engineering investigation. Additionally, the focus on representation quality over mechanism sidesteps the question of whether hybrid approaches—combining well-engineered preprocessing with modern neural retrievers—might yield even better results.
Research Context
This work directly critiques the implicit assumption in recent RAG research that neural retrievers (dense embeddings, multimodal models) are the primary source of improvements over classical methods like BM25. It builds on a lineage of papers questioning whether architectural complexity is always necessary in NLP, echoing findings in distillation and pruning literature. The paper contributes to a growing body of work on decomposing end-to-end system improvements to identify the true source of gains, similar to ablation studies in other domains. It opens a research direction around 'representation engineering' as a distinct discipline from algorithm design, suggesting that future RAG research should include systematic preprocessing baselines before assuming neural approaches are necessary.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::