RAG Systems

Retrieval-Augmented Generation (RAG) is the most common architecture for production LLM applications. If you are interviewing for an ML engineer, AI engineer, or applied scientist role, RAG questions are nearly guaranteed. This chapter covers the full pipeline - from chunking strategies through evaluation - with the depth interviewers expect.

Why RAG?

LLMs have two fundamental limitations that RAG addresses:

1. Knowledge cutoff: The model's parametric knowledge is frozen at the training data cutoff. It cannot answer questions about recent events or proprietary data.
2. Hallucination: When the model does not know the answer, it generates plausible-sounding but incorrect text rather than admitting ignorance.

RAG solves both by retrieving relevant documents at inference time and grounding the model's response in external evidence.

60-Second Answer

"RAG augments an LLM with an external knowledge base. At query time, we retrieve relevant documents, inject them into the context, and let the model generate an answer grounded in the retrieved evidence. This reduces hallucination, enables access to private or recent data, and is cheaper than fine-tuning for knowledge updates."

The RAG Pipeline

The pipeline has two phases:

Offline (indexing):

1. Load documents from your knowledge base
2. Split documents into chunks
3. Embed each chunk into a vector
4. Store vectors in a vector database

Online (query):

1. Embed the user's query
2. Retrieve the top-K most similar chunks
3. (Optional) Rerank retrieved chunks
4. Construct a prompt with the query and retrieved context
5. Generate the answer with an LLM

Chunking Strategies

Chunking is the first and most underestimated step. Poor chunking destroys retrieval quality regardless of how good your embedding model is.

Common Approaches

Strategy	How It Works	Pros	Cons
Fixed-size	Split every $N$ tokens with overlap	Simple, predictable	Breaks mid-sentence, ignores structure
Sentence-based	Split on sentence boundaries	Preserves meaning	Sentences vary wildly in information density
Paragraph-based	Split on paragraph breaks	Natural semantic units	Paragraphs can be too long or too short
Recursive character	Try large splits first, recurse to smaller if too big	Respects structure hierarchy	Requires tuning of separator hierarchy
Semantic chunking	Split when embedding similarity between consecutive sentences drops	Chunks are semantically coherent	Slower, requires embedding at chunk time
Document-aware	Use document structure (headers, sections, tables)	Preserves document logic	Requires parsing (PDF, HTML, Markdown)

Chunk Size Selection

The optimal chunk size depends on your use case:

• Small chunks (100-256 tokens): Better for precise factual retrieval ("What is the capital of France?"). Higher recall but more chunks needed.
• Medium chunks (256-512 tokens): Good default for most applications. Balances precision and context.
• Large chunks (512-1024 tokens): Better for complex reasoning that requires more context. Lower recall but each chunk is more informative.

Overlap: Typically 10-20% of chunk size. Overlap prevents information loss at chunk boundaries but increases storage and can introduce redundancy in retrieval results.

Common Trap

Many candidates use a fixed chunk size without considering the content. A 256-token chunk of dense technical documentation is very different from a 256-token chunk of conversational text. The right answer usually involves document-aware chunking that respects the natural structure of your data.

Metadata Enrichment

Attach metadata to each chunk to enable filtered retrieval:

• Source document (title, URL, date)
• Section hierarchy (chapter, section, subsection)
• Document type (FAQ, tutorial, API reference, etc.)
• Entity tags (product names, dates, authors)

This enables hybrid queries: "Find chunks about pricing from the 2024 documentation."

Embedding Models

The embedding model converts text into dense vectors that capture semantic meaning. The quality of your embeddings directly determines retrieval quality.

Popular Embedding Models

Model	Dimensions	Max Tokens	Strengths
OpenAI text-embedding-3-small	1536	8191	Good quality, low cost, easy API
OpenAI text-embedding-3-large	3072	8191	Best OpenAI quality, supports dimension reduction
Cohere embed-v3	1024	512	Multi-language, compression support
BGE-large-en-v1.5	1024	512	Strong open-source, MIT license
E5-mistral-7b	4096	32768	Long-context, instruction-following embeddings
GTE-Qwen2	1536	8192	Top MTEB scores, open-source
Nomic-embed-text	768	8192	Long-context, fully open-source

Key Concepts

Bi-encoder architecture: Both query and document are embedded independently. At search time, you only need to embed the query - document embeddings are precomputed. This is what makes vector search fast.

Instruction-tuned embeddings: Modern models like E5 and GTE accept task-specific prefixes:

• Query: "query: What is the capital of France?"
• Document: "passage: Paris is the capital and largest city of France."

This asymmetric encoding improves retrieval accuracy because queries and documents are written differently.

Matryoshka Representation Learning (MRL): Embeddings trained so that truncating to fewer dimensions preserves most quality. OpenAI's text-embedding-3 supports this - you can use 256 dimensions instead of 1536 for a 6x storage reduction with only minor quality loss.

Interviewer's Perspective

Interviewers want to hear you discuss the tradeoffs: embedding dimension (quality vs. storage/speed), open-source vs. API (cost vs. control), and how to benchmark embedding models for your specific domain (MTEB is general-purpose but may not reflect your data distribution).

Evaluating Embedding Quality

Use retrieval benchmarks on your actual data, not just MTEB leaderboard scores:

1. Create a test set of (query, relevant_document) pairs from your domain
2. Embed all documents and queries
3. Measure Recall@K, MRR, and NDCG
4. Compare multiple models - the MTEB winner may not be the best for your domain

Vector Databases

Vector databases store embeddings and support efficient approximate nearest neighbor (ANN) search.

Comparison

Database	Type	Indexing	Filtering	Hosting	Best For
Pinecone	Managed	Proprietary	Metadata filters	Cloud only	Production, zero ops
Weaviate	Open-source	HNSW	GraphQL filters	Self-hosted or cloud	Hybrid search, multimodal
Chroma	Open-source	HNSW	Metadata filters	Embedded or server	Prototyping, local dev
pgvector	Extension	IVFFlat, HNSW	Full SQL	Any Postgres	Teams with existing Postgres
Qdrant	Open-source	HNSW	Rich filtering	Self-hosted or cloud	High-performance filtering
Milvus	Open-source	IVFFlat, HNSW, DiskANN	Attribute filters	Self-hosted or cloud	Large-scale, enterprise
FAISS	Library	IVF, HNSW, PQ	None (manual)	In-process	Research, custom pipelines

Indexing Algorithms

HNSW (Hierarchical Navigable Small World):

• Most popular for production use
• Build time: $O(N \log N)$
• Query time: $O(\log N)$
• Memory: stores full vectors + graph structure
• Tradeoff: ef_construction (build quality) and ef_search (query quality vs. speed)

IVFFlat (Inverted File with Flat quantization):

• Partitions vectors into $N_{\text{lists}}$ clusters using k-means
• Query searches only the closest $N_{\text{probe}}$ clusters
• Lower memory than HNSW
• Tradeoff: $N_{\text{probe}}$ controls recall vs. speed

Product Quantization (PQ):

• Compresses vectors by splitting into subvectors and quantizing each independently
• Dramatically reduces memory (e.g., 1536-dim float32 at 6KB goes to under 200 bytes)
• Some accuracy loss - use for large-scale with reranking

Company Variation

Startups typically use Pinecone (zero ops) or Chroma (simple, local). Mid-size companies often use pgvector (if already on Postgres) or Qdrant. Large companies use Milvus or Weaviate for scale and flexibility. Research teams use FAISS directly.

Hybrid Search

Combining dense vector search with sparse keyword search (BM25) often outperforms either alone:

$$\text{score}(q, d) = \alpha \cdot \text{dense\_score}(q, d) + (1 - \alpha) \cdot \text{sparse\_score}(q, d)$$

where $\alpha \in [0, 1]$ controls the balance. Typical values are $\alpha \in [0.5, 0.8]$.

Why hybrid works: Dense search captures semantic similarity ("automobile" matches "car") but can miss exact keyword matches. Sparse search captures exact matches but misses semantic relationships. Together, they cover both cases.

Weaviate, Qdrant, and Pinecone natively support hybrid search. For pgvector, you can combine with PostgreSQL full-text search.

Retrieval and Reranking

Retrieval Parameters

• Top-K: How many chunks to retrieve. Typical values: K=5 to K=20. Higher K increases recall but also noise.
• Similarity threshold: Minimum cosine similarity to include a result. Prevents returning irrelevant chunks when the query has no good matches.
• Maximum Marginal Relevance (MMR): Diversifies results by penalizing chunks that are similar to already-selected chunks:

$$\text{MMR} = \arg\max_{d_i \in R \setminus S} \left[ \lambda \cdot \text{sim}(q, d_i) - (1 - \lambda) \cdot \max_{d_j \in S} \text{sim}(d_i, d_j) \right]$$

where $S$ is the set of already-selected documents, $R$ is the candidate set, and $\lambda$ balances relevance vs. diversity.

Reranking

Reranking is a second-stage retrieval step that uses a more powerful (but slower) model to re-score the top-K results from the initial retrieval.

Why reranking works: Bi-encoders embed query and document independently - they cannot model fine-grained interactions. Cross-encoders process the query-document pair jointly, enabling attention between query and document tokens. This is much more accurate but too slow for searching millions of documents.

Popular rerankers:

• Cohere Rerank: API-based, easy to integrate, strong quality
• bge-reranker-v2-m3: Open-source, multilingual, competitive with Cohere
• cross-encoder/ms-marco-MiniLM-L-6-v2: Lightweight, fast, good for prototyping
• RankLLM / RankGPT: Uses an LLM to rerank - more expensive but sometimes more accurate

Typical pipeline:

1. Retrieve top-100 with bi-encoder (fast ANN search)
2. Rerank to top-5 with cross-encoder (accurate but slow)
3. Feed top-5 into the LLM context

60-Second Answer

"Reranking is a two-stage retrieval pattern. The first stage uses a fast bi-encoder to find the top 50-100 candidates from millions of documents. The second stage uses a slower but more accurate cross-encoder to re-score and select the final top 5-10 for the LLM context. This gives you the speed of vector search with the accuracy of cross-attention."

Advanced RAG Techniques

HyDE (Hypothetical Document Embeddings)

Instead of embedding the user's query directly, generate a hypothetical answer and embed that instead.

Why it works: Queries and documents are written in different styles. A question like "What causes diabetes?" is semantically distant from a passage explaining diabetes mechanisms. The hypothetical answer bridges this gap because it is written in the same style as the target documents.

Tradeoff: Adds one LLM call per query (latency + cost). Works best for factual questions; less useful for vague or exploratory queries.

Multi-Query Retrieval

Generate multiple reformulations of the user's query and retrieve for each, then merge results.

Example: User asks "How does RAG work?"

• Query 1: "How does retrieval augmented generation work?"
• Query 2: "RAG pipeline architecture components"
• Query 3: "Combining vector search with LLM generation"

Retrieve for all three, deduplicate, and rerank. This captures more relevant documents because different phrasings match different chunks.

Parent-Child Chunking

Split documents into small chunks (children) for precise retrieval, but return the larger chunk (parent) for context.

Setup:

1. Create large chunks (e.g., 2000 tokens) - these are parents
2. Split each parent into small chunks (e.g., 200 tokens) - these are children
3. Embed and index only the children
4. When a child chunk is retrieved, return the parent chunk to the LLM

Why it works: Small chunks are better for precise retrieval (the embedding focuses on a specific fact). But the LLM needs surrounding context to generate a good answer. Parent-child gives you both.

Contextual Compression

After retrieval, use an LLM to extract only the relevant portions of each chunk.

Before compression:

"The company was founded in 2015 by John Smith. It is headquartered in San Francisco. The company's revenue in 2023 was $50 million, representing a 40% year-over-year growth. The company has 500 employees across 3 offices."

Query: "What was the company's revenue?"

After compression:

"The company's revenue in 2023 was $50 million, representing a 40% year-over-year growth."

This reduces noise in the context window and allows you to include more chunks.

Sentence Window Retrieval

Similar to parent-child but at the sentence level:

1. Embed individual sentences
2. When a sentence is retrieved, expand the window to include $\pm N$ surrounding sentences
3. Return the expanded window to the LLM

Agentic RAG

Use an LLM as an agent that decides how to retrieve, not just what to retrieve:

The agent can:

• Decompose complex queries into sub-queries
• Choose between different retrieval strategies (vector search, keyword search, SQL query)
• Evaluate retrieved results and decide if more retrieval is needed
• Combine information from multiple retrievals

RAG Evaluation

Evaluating RAG systems is harder than evaluating standalone LLMs because you need to assess both retrieval quality and generation quality.

The RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment) defines four key metrics:

Metric	What It Measures	How It Works
Context Precision	Are the retrieved chunks relevant to the query?	Fraction of retrieved chunks that contain relevant information
Context Recall	Did we retrieve all the information needed?	Fraction of answer claims supported by at least one retrieved chunk
Faithfulness	Is the answer grounded in the retrieved context?	Fraction of answer claims that can be attributed to the context
Answer Relevance	Does the answer address the original query?	Semantic similarity between the query and the answer

Building an Evaluation Dataset

1. Seed questions: Write 50-100 question-answer pairs from your knowledge base
2. Synthetic expansion: Use an LLM to generate variations and harder questions
3. Ground truth contexts: For each question, annotate which chunks contain the answer
4. Negative examples: Include questions that cannot be answered from the knowledge base (to test abstention)

Failure Mode Analysis

Failure Mode	Symptom	Root Cause	Fix
Hallucination	Answer contains facts not in context	LLM ignores context or fills gaps	Improve faithfulness prompt, add citation requirements
Retrieval failure	Relevant chunks not retrieved	Bad chunking, wrong embedding model, or missing data	Improve chunking, evaluate embedding models, check coverage
Context overflow	Answer misses key info despite good retrieval	Too many chunks dilute signal	Rerank, compress, use smaller top-K
Wrong abstraction level	Answer too detailed or too high-level	Chunk size mismatch	Adjust chunk size or use parent-child
Stale data	Answer uses outdated information	Knowledge base not updated	Implement refresh pipeline, add recency signals

Instant Rejection

Saying you would evaluate a RAG system using only BLEU/ROUGE or perplexity is a red flag. These metrics measure surface-level text similarity and are largely meaningless for RAG. Interviewers expect you to discuss retrieval-specific metrics (recall@K, MRR) and generation-specific metrics (faithfulness, answer relevance).

RAG vs. Fine-Tuning Decision Framework

One of the most common interview questions: "When should you use RAG vs. fine-tuning?"

Decision Matrix

Factor	RAG	Fine-Tuning	Both
Knowledge updates frequently	Strong choice	Expensive to retrain	---
Need citations/sources	Strong choice	Not naturally supported	---
Proprietary knowledge base	Strong choice	Risk of memorization	---
Specific output format/style	Weak	Strong choice	---
Domain-specific reasoning	Moderate	Strong choice	Best approach
Latency constraints	Adds retrieval latency	No added latency	---
Cost sensitivity	Longer prompts cost more per query	One-time training cost	---
Data volume is small (under 1K examples)	Strong choice	Insufficient for fine-tuning	---
Need to follow complex instructions	Moderate	Strong choice	---

60-Second Answer

"RAG is for knowledge - giving the model access to information it was not trained on. Fine-tuning is for behavior - teaching the model a new style, format, or reasoning pattern. Most production systems use RAG first (it is faster to iterate, cheaper to maintain, and naturally supports citations), and add fine-tuning only when prompting plus RAG is insufficient for the behavioral requirements."

Production RAG Considerations

Latency Optimization

Component	Typical Latency	Optimization
Embedding query	20-50ms	Cache frequent queries, use smaller model
Vector search	10-50ms	HNSW tuning, reduce dimension with MRL
Reranking	100-300ms	Batch reranking, lighter reranker model
LLM generation	500-3000ms	Streaming, shorter context, faster model
Total	630-3400ms	Target: under 2 seconds for interactive use

Caching Strategies

1. Semantic cache: Embed the query; if a similar query was recently answered, return the cached answer. Use cosine similarity threshold (e.g., 0.95).
2. Chunk cache: Cache frequently retrieved chunks in memory to avoid database round-trips.
3. Result cache: Cache the final LLM response for exact query matches (useful for FAQ-style queries).

Monitoring

Track these metrics in production:

• Retrieval hit rate: Fraction of queries where at least one relevant chunk is retrieved
• Context utilization: Does the LLM actually use the retrieved context? (measure via citation rate)
• User satisfaction: Thumbs up/down, follow-up questions (indicates incomplete answers)
• Latency percentiles: p50, p95, p99 for each pipeline stage
• Cost per query: Embedding + vector DB + reranking + LLM tokens

Practice Problems

Problem 1: Chunking Strategy

You are building a RAG system over a company's internal documentation, which includes API references, tutorials, FAQs, and meeting notes. Design a chunking strategy.

Hint 1 - Direction

Different document types have different structures. A one-size-fits-all chunking strategy will be suboptimal.

Hint 2 - Insight

Use document-aware chunking: parse each document type based on its natural structure. API references should be chunked per endpoint, tutorials per section, FAQs per question-answer pair, and meeting notes per agenda item.

Hint 3 - Full Solution

Document-type-specific chunking:

Document Type	Chunking Strategy	Chunk Size	Metadata
API reference	One chunk per endpoint (description + params + examples)	Variable (200-800 tokens)	Endpoint name, HTTP method, version
Tutorials	One chunk per section/subsection	300-500 tokens with overlap	Tutorial title, section hierarchy, difficulty
FAQs	One chunk per Q&A pair	Variable (50-500 tokens)	Category, related questions
Meeting notes	One chunk per agenda item or decision	200-400 tokens	Date, participants, project

Additional enrichment:

• Extract entities (product names, API names) and add as metadata for filtered search
• Generate a summary of each chunk using an LLM and store alongside the chunk for multi-representation retrieval
• Create cross-references: link related chunks across document types

Evaluation:

• Create test queries representative of each document type
• Measure recall@5 for each document type separately
• Iterate on chunk size and overlap based on retrieval performance

Scoring rubric:

Grade	Criteria
Strong Hire	Proposes document-type-specific chunking, discusses metadata enrichment, mentions evaluation strategy, considers cross-document relationships.
Lean Hire	Recognizes need for different strategies per document type. Uses reasonable chunk sizes with overlap.
No Hire	Uses fixed-size chunking for all document types. Does not discuss metadata or evaluation.

Problem 2: RAG Failure Debugging

Your RAG system has a faithfulness score of 0.6 (on a 0-1 scale). Users report that the chatbot sometimes makes claims not supported by the retrieved documents. How do you diagnose and fix this?

Hint 1 - Direction

Low faithfulness means the LLM is generating content not grounded in the retrieved context. This could be a retrieval problem (wrong context) or a generation problem (ignoring context).

Hint 2 - Insight

Separate diagnosis: First check if the correct information is in the retrieved context (context recall). If yes, the generation is the problem. If no, the retrieval is the problem. They require different fixes.

Hint 3 - Full Solution

Step 1: Diagnose retrieval vs. generation

• Sample 50 low-faithfulness examples
• For each, check: Is the correct information in the retrieved chunks?
  • If No (retrieval failure): Fix chunking, embedding model, or top-K
  • If Yes (generation failure): The LLM is ignoring or contradicting the context

Step 2: Fix retrieval failures

• Check chunk coverage: Does the knowledge base contain the answer?
• Evaluate embedding quality on domain-specific test set
• Try hybrid search (BM25 + dense) for keyword-sensitive queries
• Add reranking to filter irrelevant chunks

Step 3: Fix generation failures

• Strengthen the system prompt: "Only answer based on the provided context. If the context does not contain the answer, say so."
• Add explicit citation requirements: "Support each claim with a [Source N] reference."
• Reduce context noise: Fewer, more relevant chunks (aggressive reranking)
• Use a more instruction-following model
• Post-generation verification: Use a second LLM call to check faithfulness

Step 4: Systemic improvements

• Add a "confidence score" or "no answer" option so the system can abstain
• Implement a feedback loop: When users flag incorrect answers, add these as negative examples
• A/B test changes and measure faithfulness improvement

Scoring rubric:

Grade	Criteria
Strong Hire	Separates retrieval vs. generation diagnosis, proposes fixes for both, includes systematic evaluation and monitoring. Mentions citation requirements and abstention.
Lean Hire	Identifies the retrieval/generation distinction. Proposes reasonable fixes for one or both.
No Hire	Proposes only "use a better model" or "add more data" without diagnosis. Does not distinguish retrieval from generation failures.

Problem 3: Vector Database Selection

Your team is building a RAG system for a legal firm with 10 million document chunks. The system needs to support metadata filtering (by case type, jurisdiction, date range) and handle 100 concurrent queries per second. Which vector database would you recommend and why?

Hint 1 - Direction

At 10M chunks with rich metadata filtering and high concurrency, you need a database that excels at filtered ANN search and can scale horizontally.

Hint 2 - Insight

The key constraint is the combination of metadata filtering and high throughput. Some vector databases degrade significantly when combining vector search with filters. Consider whether you need managed vs. self-hosted based on the firm's compliance requirements.

Hint 3 - Full Solution

Recommendation: Qdrant or Weaviate (self-hosted) or Pinecone (managed)

Analysis:

Option	Filtering	Scale	Ops Burden	Compliance
Qdrant	Excellent (payload filtering before ANN)	10M+ easily	Moderate	Full control
Weaviate	Good (GraphQL filters)	10M+ with sharding	Moderate	Full control
Pinecone	Good (metadata filters)	10M+ serverless	None	SOC 2, but data leaves premise
pgvector	Limited at scale	Struggles above 5M	Low if Postgres exists	Full control

For a legal firm, I would recommend Qdrant (self-hosted):

• Legal data is highly sensitive - self-hosting keeps data on-premise
• Qdrant's filtering is applied before ANN search (not post-filter), so it scales well with complex filters
• HNSW index handles 10M chunks efficiently
• 100 QPS is achievable with a single node at this scale

Architecture:

• 3-node Qdrant cluster for high availability
• Segment collections by document type or jurisdiction for faster filtered queries
• Use scalar quantization (SQ) to reduce memory: 10M chunks with 1024-dim at FP32 is ~40 GB; SQ reduces to ~10 GB
• Deploy behind a load balancer with connection pooling

Scoring rubric:

Grade	Criteria
Strong Hire	Considers compliance (data sensitivity), evaluates filtering performance (pre-filter vs. post-filter), proposes concrete architecture with sizing estimates.
Lean Hire	Makes a reasonable recommendation with justification. Considers at least 2-3 options.
No Hire	Recommends a database without justification or ignores the filtering/compliance requirements.

Interview Cheat Sheet

Topic	Key Fact	Why It Matters
RAG purpose	Ground LLM in external knowledge, reduce hallucination	The "why" behind RAG
Chunking	Document-aware chunking beats fixed-size	First decision, high impact
Chunk size	256-512 tokens is a good default	Balances precision and context
Embedding models	Bi-encoders for search, cross-encoders for reranking	Two-stage retrieval pattern
Hybrid search	Dense + sparse (BM25) outperforms either alone	Covers semantic and keyword gaps
Reranking	Cross-encoder over top-K from bi-encoder	Major accuracy boost, moderate cost
HyDE	Embed hypothetical answer instead of query	Bridges query-document style gap
Parent-child chunks	Small chunks for retrieval, large for context	Best of both worlds
RAGAS metrics	Precision, recall, faithfulness, relevance	Standard RAG evaluation framework
RAG vs. fine-tuning	RAG for knowledge, fine-tuning for behavior	Most common interview question
Vector DB indexing	HNSW is the default for production	$O(\log N)$ query time
Semantic cache	Cache answers for similar queries	Critical for latency and cost
Failure modes	Retrieval failure vs. generation failure	Different diagnosis, different fix

Spaced Repetition Checkpoints

Day 0 (Today)

• Draw the full RAG pipeline from query to response
• Name three chunking strategies and when to use each
• What is the difference between a bi-encoder and a cross-encoder?

Day 3

• Explain HyDE, multi-query retrieval, and parent-child chunking
• List the four RAGAS metrics and what each measures
• When would you choose RAG over fine-tuning? Give three scenarios.

Day 7

• Design a production RAG system for a specific domain (pick one: legal, medical, customer support)
• Compare three vector databases for a 10M-document use case
• How would you diagnose and fix a faithfulness score of 0.6?

Day 14

• Explain hybrid search: why it works, how to tune $\alpha$, and when it matters
• Design a RAG evaluation pipeline with synthetic test generation
• Walk through the latency breakdown of a RAG query and identify optimization opportunities

Day 21

• Teach someone the full RAG stack from chunking through evaluation
• Design an agentic RAG system that handles multi-step queries
• Propose a monitoring and alerting strategy for a production RAG system