RAG vs Long Context - When to Use Each
The New Architecture Decision
For three years, the RAG pattern was the only viable answer to the question "how do I give an LLM access to a large knowledge base?" The context windows were too small to fit more than a few documents, so you retrieved the most relevant chunks and fed those in.
Then Gemini 1.5 Pro launched with 1M token context. Claude 3 launched with 200K. GPT-4o with 128K. Suddenly the question became: why retrieve at all? If the model can read everything, why not give it everything?
The question is more nuanced than it appears. The answer is almost never "always use long context" or "always use RAG." It depends on specific tradeoffs that are computable - cost, latency, accuracy - and on properties of your specific use case that determine which tradeoffs matter most.
This lesson gives you the framework to make that decision rigorously.
The Core Tradeoffs
Cost
Long-context models charge per input token. At 128K tokens per query, the economics are dramatically different from RAG:
def compare_retrieval_cost(
corpus_size_tokens: int,
queries_per_day: int,
rag_top_k_tokens: int = 4_000, # Typical RAG chunk retrieval
long_context_tokens: int = 128_000, # Full context stuffing
embedding_cost_per_million: float = 0.02, # text-embedding-3-large
# API costs (USD/million tokens, approximate 2024 prices):
gpt4o_input_cost: float = 5.0,
gpt4o_mini_input_cost: float = 0.15,
claude35_input_cost: float = 3.0,
output_tokens: int = 500,
gpt4o_output_cost: float = 15.0,
) -> dict:
"""
Compare daily costs of RAG vs long-context stuffing.
"""
def api_cost(input_tokens, input_price, output_tokens, output_price):
return (input_tokens / 1e6 * input_price +
output_tokens / 1e6 * output_price) * queries_per_day
# RAG costs
# Embedding cost is one-time for indexing (amortized here to zero for simplicity)
rag_gpt4o = api_cost(rag_top_k_tokens, gpt4o_input_cost, output_tokens, gpt4o_output_cost)
rag_gpt4o_mini = api_cost(rag_top_k_tokens, gpt4o_mini_input_cost, output_tokens, 0.6)
# Long-context costs
lc_gpt4o = api_cost(long_context_tokens, gpt4o_input_cost, output_tokens, gpt4o_output_cost)
lc_claude = api_cost(long_context_tokens, claude35_input_cost, output_tokens, 15.0)
results = {
"RAG + GPT-4o (top-k=4K)": rag_gpt4o,
"RAG + GPT-4o-mini (top-k=4K)": rag_gpt4o_mini,
"Long Context GPT-4o (128K)": lc_gpt4o,
"Long Context Claude-3.5 (128K)": lc_claude,
}
print(f"Daily cost comparison ({queries_per_day:,} queries/day):")
print(f"{'Method':<35} | Daily Cost | Annual Cost")
print("-" * 65)
for method, daily in results.items():
annual = daily * 365
print(f"{method:<35} | ${daily:>8,.2f} | ${annual:>12,.2f}")
return results
# Example: 1000 queries/day
compare_retrieval_cost(
corpus_size_tokens=50_000_000, # 50M token knowledge base
queries_per_day=1_000,
)
# Typical output:
# Daily cost comparison (1,000 queries/day):
# Method | Daily Cost | Annual Cost
# -----------------------------------------------------------------
# RAG + GPT-4o (top-k=4K) | $ 27.25 | $ 9,947
# RAG + GPT-4o-mini (top-k=4K) | $ 0.90 | $ 329
# Long Context GPT-4o (128K) | $ 647.50 | $ 236,338
# Long Context Claude-3.5 (128K) | $ 391.00 | $ 142,715
#
# Conclusion: RAG is 24-720× cheaper per query depending on model choice
The cost difference is not marginal - it's 1-3 orders of magnitude. For high-volume applications, long-context stuffing is simply not viable economically unless you're running local models.
Latency
import time
def estimate_latency(
input_tokens: int,
output_tokens: int,
# Time-to-first-token (TTFT) grows with input length
# This scales roughly linearly with input for prefill-dominated models
ttft_ms_per_1k_input: float = 80, # GPT-4o approximate
output_tokens_per_second: float = 50, # Approximate for GPT-4o
# RAG additional latency
vector_search_ms: float = 50, # Typical Pinecone/Weaviate latency
reranking_ms: float = 200, # Cross-encoder reranking (optional)
chunking_ms: float = 10, # Tokenization overhead
) -> dict:
"""Estimate end-to-end latency for RAG vs long context."""
# LLM inference latency
llm_ttft = ttft_ms_per_1k_input * (input_tokens / 1000)
output_latency = (output_tokens / output_tokens_per_second) * 1000
total_llm_ms = llm_ttft + output_latency
rag_overhead = vector_search_ms + reranking_ms + chunking_ms
return {
"RAG total ms": rag_overhead + (ttft_ms_per_1k_input * 4) + output_latency,
"RAG overhead ms": rag_overhead,
"Long context total ms": total_llm_ms,
"RAG vs LongCtx ratio": (rag_overhead + (ttft_ms_per_1k_input * 4) + output_latency) / total_llm_ms,
}
# Compare at different context sizes
for ctx in [4_000, 16_000, 32_000, 128_000]:
result = estimate_latency(input_tokens=ctx, output_tokens=500)
rag_ms = result["RAG total ms"]
lc_ms = result["Long context total ms"]
winner = "RAG faster" if rag_ms < lc_ms else "LongCtx faster"
print(f" {ctx:>8,} tokens: RAG={rag_ms:>6,.0f}ms, LongCtx={lc_ms:>6,.0f}ms | {winner}")
# Output:
# 4,000 tokens: RAG= 830ms, LongCtx= 330ms | LongCtx faster
# 16,000 tokens: RAG= 830ms, LongCtx= 960ms | RAG faster
# 32,000 tokens: RAG= 830ms, LongCtx= 2,560ms | RAG faster
# 128,000 tokens: RAG= 830ms, LongCtx= 10,240ms | RAG faster
The latency crossover: At short contexts (4-8K), long-context inference can actually be faster than RAG because there's no retrieval overhead. Beyond 16-32K tokens, RAG becomes faster because the time-to-first-token grows with context length while retrieval latency is constant.
Accuracy
Accuracy is harder to quantify generically because it depends on:
- Retrieval quality (how often does the top-k actually contain the relevant information?)
- Lost-in-middle effects (how reliably does the model use middle-context information?)
- Knowledge base coverage (is everything retrievable, or is some information only in the raw corpus?)
def accuracy_model(
retrieval_recall: float = 0.92, # P(relevant chunk in top-k)
in_context_recall: float = 0.88, # P(model uses info | it's in context)
long_context_retrieval: float = 0.75, # P(model uses info at middle position)
synthesis_advantage: float = 0.15, # Long-context advantage for synthesis tasks
) -> dict:
"""
Simple model of task-specific accuracy for RAG vs long context.
For retrieval tasks:
- RAG accuracy ~ retrieval_recall × in_context_recall
- LongCtx accuracy ~ long_context_retrieval (assuming info is always present)
For synthesis tasks (aggregate info across many documents):
- RAG accuracy ~ retrieval_recall × in_context_recall × (correlation factor)
- LongCtx accuracy ~ long_context_retrieval + synthesis_advantage
"""
rag_retrieval = retrieval_recall * in_context_recall
lc_retrieval = long_context_retrieval # middle positions hurt
rag_synthesis = retrieval_recall * in_context_recall * 0.70 # harder to retrieve all facts
lc_synthesis = long_context_retrieval + synthesis_advantage # can see everything
return {
"Task: Single-fact retrieval": {
"RAG": round(rag_retrieval, 3),
"Long context": round(lc_retrieval, 3),
"Winner": "RAG" if rag_retrieval > lc_retrieval else "Long context",
},
"Task: Multi-hop reasoning": {
"RAG": round(rag_retrieval * 0.8, 3), # Multiple retrievals, each with recall penalty
"Long context": round(lc_retrieval * 0.85, 3),
"Winner": "Depends",
},
"Task: Synthesis across many docs": {
"RAG": round(rag_synthesis, 3),
"Long context": round(lc_synthesis, 3),
"Winner": "Long context" if lc_synthesis > rag_synthesis else "RAG",
},
}
results = accuracy_model()
for task, comparison in results.items():
print(f"\n{task}:")
print(f" RAG: {comparison['RAG']:.1%}")
print(f" Long context: {comparison['Long context']:.1%}")
print(f" Winner: {comparison['Winner']}")
When Long Context Wins
Long context is the right choice when:
1. The task requires synthesizing across many documents
Summarization, comparative analysis, trend identification across a corpus - these tasks require seeing many sources simultaneously. RAG with top-k retrieval often misses the 11th or 15th most relevant document, which is fine for single-fact retrieval but bad for synthesis.
2. The relevant information cannot be predicted in advance
Some queries require information whose relevance is only apparent after reading many documents. Retrieval assumes you can identify what to retrieve before retrieval - but if the answer is "compare document A's financial projections to document B's market size, and note where document C's risks intersect," no query embedding captures this before reading.
3. The corpus is small enough to fit in context
If your entire knowledge base is 50 pages (25K tokens), skip RAG entirely. Just put everything in context. The complexity of a retrieval pipeline is not worth it when the entire corpus fits comfortably.
4. Coherence across the full document matters
For tasks like "review this entire codebase for security vulnerabilities" or "identify inconsistencies in this legal agreement," the model needs to see all of it - not retrieved fragments. Coherence and cross-reference matter.
When RAG Wins
RAG is the right choice when:
1. The knowledge base is large - much larger than any context window
A 10-million-token corpus (a medium-sized company's documentation) cannot fit in any current context window. RAG is the only option. And even if context windows reach 10M tokens, the economics of stuffing that much text per query will remain prohibitive.
2. The knowledge base changes frequently
Updating a vector index is fast (add/remove documents). Re-processing a model to incorporate new knowledge requires fine-tuning or continuing pre-training. For frequently updated content (news, product docs, prices), RAG is the natural architecture.
3. Queries are specific and retrievable
When users ask "what is the return policy for product X?", the answer is in exactly one place, and a good retrieval system will find it. There's no benefit to loading the entire product manual when the relevant chunk is well-defined.
4. Cost and latency are constraints
For high-volume, latency-sensitive, or budget-constrained deployments, RAG with a smaller model (GPT-4o-mini, Llama-3-8B) is often the only economically viable architecture.
The Hybrid Approach - Best of Both
The most robust production architecture combines RAG and long context:
This hybrid approach:
- Uses retrieval to filter a large corpus (millions of tokens → hundreds)
- Uses a reranker to get the most relevant subset (hundreds → tens)
- Uses a longer context window to give the LLM more complete information
- Uses strategic placement to mitigate lost-in-middle effects
from typing import Any
import numpy as np
class HybridRAGPipeline:
"""
Hybrid RAG pipeline: vector retrieval + reranking + long-context LLM.
Combines the corpus coverage of RAG with the synthesis ability
of long-context models.
"""
def __init__(
self,
vector_store, # Pinecone, Weaviate, Chroma, etc.
reranker, # Cross-encoder reranker (e.g., ms-marco-MiniLM)
llm_client, # OpenAI, Anthropic, etc.
model_name: str = "gpt-4o",
initial_top_n: int = 100, # Initial retrieval count
reranked_top_k: int = 20, # After reranking
max_context_tokens: int = 50_000,
):
self.vector_store = vector_store
self.reranker = reranker
self.llm_client = llm_client
self.model_name = model_name
self.initial_top_n = initial_top_n
self.reranked_top_k = reranked_top_k
self.max_context_tokens = max_context_tokens
def retrieve(self, query: str) -> list[dict]:
"""Step 1: Vector similarity search."""
results = self.vector_store.query(
query_text=query,
top_k=self.initial_top_n,
)
return [{"text": r.text, "score": r.score, "metadata": r.metadata}
for r in results]
def rerank(self, query: str, documents: list[dict]) -> list[dict]:
"""Step 2: Cross-encoder reranking for better precision."""
scores = self.reranker.compute_scores(
query=query,
passages=[d["text"] for d in documents],
)
for doc, score in zip(documents, scores):
doc["rerank_score"] = score
# Sort by rerank score, take top-K
return sorted(documents, key=lambda x: -x["rerank_score"])[:self.reranked_top_k]
def arrange_for_retrieval(self, documents: list[dict]) -> list[dict]:
"""
Step 3: Strategic placement to mitigate lost-in-middle.
Interleaves highest-scoring documents at boundaries.
"""
n = len(documents)
result = [None] * n
front, back = 0, n - 1
for i, doc in enumerate(documents):
if i % 2 == 0:
result[front] = doc
front += 1
else:
result[back] = doc
back -= 1
return result
def build_prompt(self, query: str, documents: list[dict]) -> str:
"""Build the long-context prompt with strategic document placement."""
context_parts = []
total_tokens = 0
approx_tokens_per_char = 0.25
for i, doc in enumerate(documents):
part = f"[Document {i+1}]\n{doc['text']}"
approx_tokens = len(part) * approx_tokens_per_char
if total_tokens + approx_tokens > self.max_context_tokens:
break
context_parts.append(part)
total_tokens += approx_tokens
context = "\n\n".join(context_parts)
return (
f"You are a helpful assistant. Answer the question based on the provided documents. "
f"The answer may be in any document, including those in the middle of the list. "
f"Cite the document number when relevant.\n\n"
f"Documents:\n{context}\n\n"
f"Question: {query}\n\n"
f"Answer:"
)
def query(self, user_query: str) -> dict:
"""Run the full hybrid RAG pipeline."""
import time
timings = {}
t0 = time.time()
# Step 1: Vector retrieval
raw_docs = self.retrieve(user_query)
timings["retrieval_ms"] = (time.time() - t0) * 1000
# Step 2: Reranking
t1 = time.time()
reranked_docs = self.rerank(user_query, raw_docs)
timings["reranking_ms"] = (time.time() - t1) * 1000
# Step 3: Strategic arrangement
arranged_docs = self.arrange_for_retrieval(reranked_docs)
# Step 4: Build prompt and query LLM
t2 = time.time()
prompt = self.build_prompt(user_query, arranged_docs)
answer = self._call_llm(prompt)
timings["llm_ms"] = (time.time() - t2) * 1000
return {
"answer": answer,
"n_documents_retrieved": len(raw_docs),
"n_documents_used": len(arranged_docs),
"timings": timings,
}
def _call_llm(self, prompt: str) -> str:
"""Call the LLM with the constructed prompt."""
# Implementation depends on the specific client
response = self.llm_client.chat.completions.create(
model=self.model_name,
messages=[{"role": "user", "content": prompt}],
max_tokens=1000,
)
return response.choices[0].message.content
Decision Framework
Prompt Caching - The Third Option
Many providers now offer prompt caching: if the same long context prefix is reused across multiple queries (e.g., your 100K-token system documentation), the KV cache for that prefix is stored and reused. You pay full price on the first call; subsequent calls with the same prefix are charged at a much lower rate (Anthropic: 0.1× the regular price for cached prefix tokens).
class CachedLongContextPipeline:
"""
Long-context pipeline with prompt caching for repeated corpora.
Use when:
- Same large document/corpus is queried repeatedly
- Multiple users query the same knowledge base
- Batch queries over a single large document
"""
def __init__(self, client, model: str = "claude-3-5-sonnet-20241022"):
self.client = client
self.model = model
self._cached_context = None
self._cache_hit_count = 0
def set_context(self, context_text: str):
"""
Pre-load the long context (will be cached on first use).
Subsequent queries reuse the cached KV without re-encoding.
"""
self._cached_context = context_text
self._cache_hit_count = 0
print(f"Context loaded: ~{len(context_text) // 4:,} tokens")
def query(self, question: str) -> str:
"""
Query using the cached context.
Anthropic's API automatically uses the KV cache when the same
system prompt prefix is sent with cache_control={"type": "ephemeral"}.
"""
if self._cached_context is None:
raise ValueError("Call set_context() first.")
response = self.client.messages.create(
model=self.model,
max_tokens=1000,
system=[
{
"type": "text",
"text": f"You are a helpful assistant. Use the following documents to answer questions:\n\n{self._cached_context}",
"cache_control": {"type": "ephemeral"}, # Enable caching
}
],
messages=[{"role": "user", "content": question}],
)
# On first call: full input token cost
# On subsequent calls: cached_creation_input_tokens × 0.25 (Anthropic pricing)
usage = response.usage
self._cache_hit_count += 1
print(f" Cache hit #{self._cache_hit_count}: "
f"cached={getattr(usage, 'cache_read_input_tokens', 0):,} tokens")
return response.content[0].text
# Example: Analyze the same 100K-token document with multiple questions
pipeline = CachedLongContextPipeline(anthropic_client)
pipeline.set_context(your_100k_document)
# First query: full cost for the 100K tokens
answer1 = pipeline.query("What are the main risk factors?")
# Subsequent queries: ~10% of the token cost
answer2 = pipeline.query("What are the contractual obligations?")
answer3 = pipeline.query("What are the termination clauses?")
# Cache saves approximately $9 per 100K-token query on Claude 3.5 Sonnet
Practical Comparison Matrix
| Dimension | Pure RAG | Hybrid RAG | Long Context | Long Context + Cache |
|---|---|---|---|---|
| Corpus size limit | None | None | Context window | Context window |
| Cost per query | Low ($0.001-0.05) | Medium ($0.05-0.5) | High ($0.5-10) | Low after warmup ($0.05-1) |
| Latency | 0.5-2s | 1-3s | 2-15s | 2-15s (same) |
| Single-fact accuracy | High (with good retrieval) | Very high | High-medium | High-medium |
| Synthesis accuracy | Medium | High | Very high | Very high |
| Implementation complexity | High | Very high | Low | Medium |
| Handles corpus updates | Immediate | Immediate | Requires re-loading | Requires cache invalidation |
Common Mistakes
:::danger Don't assume retrieval recall is high without measuring it The most common RAG failure is optimistic retrieval recall assumptions. "Our embedding model is good, so we'll retrieve the right chunks" is not a measurement. For every production RAG system, measure: what fraction of test queries have the gold answer in the retrieved top-k? Values below 85% are common in practice and immediately limit maximum accuracy regardless of LLM quality. :::
:::warning Don't use the full context window just because you can "We support 128K, so let's give the model everything" is not a strategy. Models degrade at long contexts (lost-in-middle), the cost is orders of magnitude higher, and latency increases dramatically. Choose the context length that's sufficient for your task, not the maximum available. :::
:::tip Measure retrieval recall before investing in LLM optimization A common mistake in RAG systems: spending weeks optimizing prompts and model choice while ignoring that retrieval recall is 70%. If the right information isn't in the retrieved chunks, no amount of LLM improvement helps. Fix retrieval first. :::
Interview Q&A
Q: When would you choose long-context LLMs over RAG, and why?
A: Long context wins when the task requires synthesis across many documents (the model needs to see multiple sources simultaneously to form a complete answer), when the relevant information can't be identified before reading (queries that require seeing everything to know what's relevant), when the corpus is small enough to fit in context (simpler architecture), or when coherence across an entire document matters (code review, contract analysis). RAG wins when the corpus is larger than any context window, when the corpus updates frequently (easy index updates), when queries are specific and retrievable, or when cost/latency are constraints.
Q: What is the latency crossover point between RAG and long-context inference?
A: RAG has a roughly constant latency overhead from retrieval (50-100ms for vector search) plus reranking (100-300ms optional). Long-context inference latency grows linearly with input length (time-to-first-token is proportional to input tokens during the prefill phase). At approximately 8-16K input tokens, long-context inference latency equals RAG overhead. Below this, long-context can actually be faster. Above this, RAG is faster because its latency doesn't grow with corpus size. For very long contexts (128K), long-context inference time-to-first-token can exceed 10 seconds, while RAG remains under 2 seconds.
Q: How does prompt caching change the economics of long-context models?
A: Prompt caching stores the KV representations of a long, static context prefix so subsequent queries can reuse it without re-encoding the prefix. Providers like Anthropic charge approximately 10% of regular price for cached tokens, and the latency for subsequent queries is reduced (only the new query is processed, not the full context). This makes long-context economically viable for use cases where the same large document is queried repeatedly - multiple users querying the same knowledge base, batch question answering over a single document, or interactive sessions where the context remains constant across turns.
Q: Why is retrieval recall the most important metric in a RAG system?
A: Retrieval recall measures what fraction of queries have the relevant information in the retrieved top-k chunks. If recall is 80%, then 20% of queries cannot possibly be answered correctly regardless of how good the LLM is - the relevant information isn't in the context. Maximum possible accuracy is bounded by retrieval recall. This is why improving retrieval recall from 80% to 90% typically improves end-to-end accuracy more than switching to a better LLM. Common reasons for low recall: embedding model not suited to the domain, chunk size too small (answer spans chunk boundary), top-k too small, no hybrid search (dense + sparse).
Q: Describe the hybrid RAG architecture and its advantages over pure RAG or pure long context.
A: Hybrid RAG uses vector retrieval to filter a large corpus to a manageable subset (e.g., top-100 documents from 1 million), a cross-encoder reranker to identify the most relevant subset (top-20 from 100), and then gives these 20 documents to a long-context LLM. Advantages over pure RAG: better recall (more initial candidates means less chance of missing the relevant chunk), and better synthesis (20 documents in context allows the model to reason across multiple sources). Advantages over pure long context: dramatically lower cost (20 documents vs. the full corpus), lower latency, and ability to handle corpora larger than any context window.
:::tip 🎮 Interactive Playground
Visualize this concept: Try the RAG Pipeline demo on the EngineersOfAI Playground - no code required.
:::