AI Letters #01 - Why Most RAG Systems Fail in Production
RAG looks simple on paper. Retrieve relevant chunks, inject into context, generate. Ship it.
In production, it breaks in ways nobody talks about.
The Gap Between Demo and Production
Every RAG demo works. You embed your docs, run a similarity search, inject the top-k results into a prompt, and the LLM answers correctly. It's clean. It's fast. You're impressed.
Then you ship it. And within two weeks you're getting bug reports that the system confidently answers questions with outdated information, hallucinates references that don't exist in your corpus, or - worst - retrieves perfectly relevant chunks and still produces a wrong answer.
Here's what's actually happening.
Problem 1: Embedding Drift
Your embeddings are a snapshot. You embedded your docs on Day 1 using model text-embedding-ada-002. Six months later, your docs have changed - new content, updated policies, deprecated endpoints - but your vector index hasn't.
What it looks like: Users ask about features that existed in v1. The retrieval finds the old chunk (high similarity score), injects it, and the LLM answers confidently about a feature that no longer exists.
Fix: Treat your vector index like a database. Track document versions. Re-embed on content change, not on a schedule.
Problem 2: Retrieval-Generation Mismatch
Top-k retrieval optimises for semantic similarity, not for answer-ability. A chunk can be highly relevant to the query but contain zero information that helps the LLM produce a correct answer.
Example: User asks "What's the maximum file size for uploads?". Top-1 retrieved chunk: "Our storage system supports enterprise-scale file operations with high throughput." Semantically relevant. Useless for the answer.
Fix: Re-rank retrieved chunks using a cross-encoder before injecting into context. Cohere Rerank, BGE-reranker, or a simple LLM-based relevance filter. The extra latency (50-100ms) is worth it.
Problem 3: Context Overflow Without Degradation Signals
When your retrieved context exceeds the model's effective attention window, the LLM doesn't throw an error. It just starts ignoring content - specifically, content in the middle of a long context (the "lost in the middle" problem, documented by Liu et al., 2023).
What it looks like: You pass 8 chunks. Chunks 1 and 8 influence the answer. Chunks 3-6 are effectively invisible. You have no idea this is happening.
Fix: Keep retrieved context under 60% of the model's context window. If you need more coverage, use a hierarchical retrieval strategy - coarse retrieval first, then fine-grained retrieval on the top result.
What a Production-Grade RAG Architecture Actually Looks Like
Query
│
├─ Query rewriting (expand abbreviations, resolve pronouns)
│
├─ Hybrid retrieval (dense + sparse / BM25)
│
├─ Cross-encoder reranking (top-20 → top-4)
│
├─ Context compression (remove redundant sentences)
│
└─ Generation with source attribution
Each step adds latency. Each step removes a failure mode. That's the tradeoff.
The Lesson
RAG is not a feature. It's an information retrieval pipeline with an LLM at the end. Every problem in classical IR - index staleness, precision vs recall, result relevance - shows up in RAG, just with a confident-sounding LLM on top that makes the failures invisible until they aren't.
Build accordingly.
AI Letters drops every week - deep engineering breakdowns for people building intelligent systems. Subscribe here.