RAG was supposed to fix the hallucination problem. Here's why it got complicated.
The gap between a demo RAG pipeline and one that works in production is larger than most people expect.
This letter was first published on AI Letters →
There's a moment every engineer hits, usually around week two of building a RAG system.
The demo worked. You chunked some PDFs, shoved them into a vector store, wired up an LLM, and it answered questions correctly. Your manager saw it and said "ship it." You were proud of it.
Then someone asked a question that crossed two documents. Or referenced something from six months ago. Or used slightly different terminology than what was in the source. And the system confidently returned something wrong.
That moment is the gap between RAG as a concept and RAG as a production system.
The promise was real
Retrieval-Augmented Generation made a simple, elegant argument: LLMs hallucinate because they have to reconstruct facts from parametric memory - the weights baked in during training. Give them the relevant documents at inference time, and they don't have to guess. They read, then answer.
It worked. The original Facebook AI paper showed clear improvements on knowledge-intensive tasks. Teams started plugging it into internal tools, customer support bots, and documentation search. The architecture - embed, retrieve, generate - fit on a single slide.
The promise was real. The problem is that the slide hid a lot of engineering.
Where it gets complicated
Chunking is a product decision, not a technical one.
Most tutorials chunk by token count - split every 512 tokens, done. In practice, the chunk boundaries determine whether the model ever sees the right context. A legal clause that starts on one chunk and ends on another is useless. A product description split from its pricing information will generate wrong answers confidently.
The right chunking strategy depends on your documents. PDFs structured as reports need different treatment than API documentation, which needs different treatment than support tickets. There's no universal answer. You have to look at your actual data and decide.
Retrieval quality and answer quality are not the same thing.
This one catches people off guard. You can build an excellent retriever - high recall, surfacing genuinely relevant chunks - and still get bad answers. Because the LLM has to synthesize across those chunks, reconcile contradictions, and figure out what's actually relevant to the question.
Top-k retrieval returns the k most similar vectors. It doesn't return the k most useful chunks for answering the question. Those are related but not identical. A chunk can be highly similar to the query embedding without containing the answer.
Semantic search has a vocabulary problem.
Vector embeddings capture meaning well when the query and the document use similar language. When they use different language for the same concept - "myocardial infarction" vs "heart attack," "churn" vs "cancellation rate," internal jargon vs customer terminology - similarity scores drop even when the documents are directly relevant.
Hybrid search (dense + sparse retrieval, i.e., vectors + BM25) helps significantly here. It's not the default setup in most tutorials. It should be.
The model forgets what it retrieved.
This is the "lost in the middle" problem, and it's well-documented. LLMs pay more attention to information at the beginning and end of the context window. Stuff buried in the middle gets underweighted. If you're stuffing 10 retrieved chunks into the prompt and the relevant one happens to land in position 5, you'll see degraded performance compared to position 1.
The fix - reranking, selecting fewer higher-quality chunks, ordering by relevance - adds latency and complexity.
Hallucination doesn't go away. It shifts.
This is the uncomfortable truth. RAG reduces hallucination on questions where the retrieved context is sufficient. But the model still has to decide when to trust the retrieved context versus its parametric memory. When the context is incomplete, it fills gaps. When the context is ambiguous, it picks an interpretation. When you ask a question that no retrieved document answers cleanly, it synthesizes something.
The hallucination problem didn't get solved. It got smaller and harder to predict.
What actually works
Teams that run RAG in production reliably share a few patterns:
Evaluation before optimization. You need a test set of real questions with ground-truth answers before you can tune anything. Without it, you're guessing. Most teams skip this at the start and spend weeks tuning into the dark.
Retrieval metrics separately from generation metrics. Measure retrieval recall (did the right chunks come back?) and generation faithfulness (did the model actually use them?) independently. When something breaks, you need to know which half failed.
Explicit fallbacks for low-confidence answers. If the retriever returns nothing above a similarity threshold, say so. "I don't have enough information to answer this reliably" is a better product experience than a confident wrong answer.
Human review of failure cases. Build a way to capture when users push back on an answer, rate it poorly, or rephrase the question. That signal is your roadmap for what to fix next.
The honest version
RAG is still the right approach for knowledge-intensive applications. It's grounded in sound reasoning and it works. But it works the way most engineering does - not by eliminating the hard problems, but by moving them somewhere more visible and tractable.
The hard problem used to be: the model doesn't know the facts. The hard problem now is: does the model know which facts to use, and does it know when it doesn't know?
That's a harder engineering problem than it looks on a slide. But it's also a more honest one.
Next letter: what the benchmarks say about long-context models - and why the benchmark isn't the thing that matters.
You're reading AI Letters - a weekly newsletter for engineers building with AI. If someone forwarded this to you, you can subscribe at engineersofai.substack.com.