The Evolution of RAG
From "look it up" to production-grade retrieval-augmented systems.
RAG started as a simple idea: if the model doesn't know the answer, look it up. Lewis et al. proved this works. Then the field spent four years making the 'look it up' part actually reliable. Each paper below fixes one specific failure mode exposed by the previous one.
Read in order. Each paper fixes a failure mode exposed by the previous one. Click any card to collapse it.
RAG — The Original Paper
Lewis, Perez, Piktus et al. (Facebook AI Research)
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Introduces the RAG pattern: a dense retriever fetches relevant passages from Wikipedia, which are concatenated with the query and fed to a seq2seq generator (BART). Two variants — RAG-Sequence and RAG-Token — are benchmarked across open-domain QA tasks.
Key insight: Generation conditioned on retrieved context outperforms pure parametric models on knowledge-intensive tasks without per-dataset fine-tuning. The retriever and generator can be jointly trained end-to-end.
Engineering note: Retrieval is global — one query, one retrieval pass at the start. No adaptation based on what the model is generating. This becomes the first failure mode to fix.
ColBERT — Multi-Vector Retrieval
Khattab & Zaharia (Stanford)
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction
Instead of one embedding per document, ColBERT produces one embedding per token. Similarity is computed as the sum of maximum inner products across all token pairs (MaxSim operation), enabling fine-grained token-level matching at scale.
Key insight: You don't have to choose between retrieval quality and retrieval speed. Late interaction — encode offline, interact at query time — gives you both. This is why ColBERT v2 is still competitive with much larger retrievers.
Engineering note: Storage cost is much higher than single-vector retrieval: one vector per token, not per document. ColBERT v2 compresses this via residual compression. Evaluate whether your retrieval quality justifies the storage overhead.
FiD — Fusion-in-Decoder
Izacard & Grave (Facebook AI Research)
Leveraging Passage Retrieval with Generative Models for Open Domain QA
Fusion-in-Decoder (FiD): encode each retrieved passage independently together with the query using the encoder, then concatenate all encoded representations and decode once. Scales to 100 retrieved passages without blowing up context.
Key insight: The bottleneck in RAG is the decoder attention, not the encoder. By encoding passages independently and fusing at decoding, you get linear scaling in the number of passages rather than quadratic. More retrieved passages = better answers, for free.
Self-RAG — Adaptive Retrieval
Asai, Wu, Wang et al. (University of Washington)
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Trains a model to decide when to retrieve, retrieve on demand, and critique its own output using special reflection tokens: [Retrieve], [IsREL], [IsSUP], [IsUSE]. The model learns that not every query needs retrieval, and that retrieved content should be verified.
Key insight: Unconditional retrieval adds latency and noise. A model that knows when to retrieve — and when to answer from memory — is faster and more accurate than one that always retrieves. The self-critique tokens are the key innovation.
Engineering note: Requires fine-tuning with the special tokens baked into the model. Not plug-and-play with arbitrary LLMs. Evaluate whether the fine-tuning cost is justified versus a simpler adaptive retrieval heuristic.
FLARE — Forward-Looking Retrieval
Jiang, Xu, Gao et al. (CMU)
FLARE: Active Retrieval Augmented Generation
FLARE generates a tentative next sentence, identifies low-confidence tokens (below a probability threshold), and if found, uses the tentative sentence as a retrieval query before committing to the output. Retrieval is triggered mid-generation, not just at the start.
Key insight: The best retrieval query is often what the model is about to say, not what the user originally asked. If the model is generating 'The capital of France is...' and is uncertain, query for that sentence — not for the original user question.
RAPTOR — Hierarchical Tree Indexing
Sarthi, Abdullah, Tuli et al. (Stanford)
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Clusters document chunks, summarizes each cluster with an LLM, then clusters and summarizes the summaries — recursively building a tree. At retrieval time, search is performed across all tree levels simultaneously, enabling both fine-grained (leaf) and high-level (root) retrieval.
Key insight: Standard RAG retrieves leaves. RAPTOR also retrieves branches and the trunk. For questions like 'What are the main themes across all these documents?', you need the upper levels. For 'What does section 3.2 say?', you need the leaves. RAPTOR gives you both.
Engineering note: Index construction is expensive — requires LLM calls per cluster. Worth it for long-document corpora where synthesis questions are common. Not worth the cost for simple factual retrieval.
CRAG — Corrective Retrieval
Yan, Gu, Chen, Chen (Various institutions)
Corrective Retrieval Augmented Generation
Adds a lightweight evaluator that scores retrieved documents for relevance. If all docs score below a threshold, falls back to web search. If mixed quality, applies knowledge refinement — stripping irrelevant sentences from chunks before injection.
Key insight: Your retriever will sometimes return garbage. Every RAG system needs a relevance gate. CRAG makes that gate systematic rather than heuristic, and gives you a fallback (web search) when your local index fails.
GraphRAG — Knowledge Graph Retrieval
Edge, Trinh, Cheng et al. (Microsoft Research)
From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Builds a knowledge graph from the document corpus: entity extraction, relationship extraction, community detection (Leiden algorithm). At query time, retrieves at the graph level — communities of related entities — rather than raw text chunks. Enables answering 'What are all the themes across this entire corpus?'
Key insight: Standard RAG is document-local. GraphRAG is corpus-global. When your users ask questions that span the entire knowledge base rather than a single passage, you need graph-level retrieval. The tradeoff: expensive indexing, transformative for the right use case.
Engineering note: Microsoft's open-source implementation is production-usable but indexing a large corpus requires substantial LLM API budget. Benchmark your corpus size and question types before committing.
What to Build After This Roadmap
You now understand every major failure mode in RAG and the paper that fixes it. A production system should combine:
Query → Rewriting → Hybrid retrieval (dense + BM25)
→ Reranking (cross-encoder) → Relevance gate (CRAG)
→ Context compression → Generation with attribution
Related