RAG Evaluation Metrics
The Silent Failure
A legal tech startup builds a RAG system to help attorneys research case precedents. The system retrieves relevant case documents and generates summaries. Automatic evaluation looks excellent: ROUGE-L of 0.45, BERTScore F1 of 0.88, latency under 2 seconds. The team ships it.
Six months in, an attorney notices something disturbing. The system has been consistently producing summaries that blend facts from multiple retrieved cases - taking the ruling from one case and the reasoning from a different case and presenting it as a coherent summary of a single precedent. Each individual sentence is plausible and well-grounded, but the combined answer attributes reasoning to cases that never said that reasoning.
ROUGE and BERTScore could not catch this because they measure lexical and semantic similarity to reference answers - and the reference answers were generated in the same problematic way. The system was faithfully reproducing the pattern of its flawed references. The failure was structural, not surfaced by any metric the team had deployed.
RAG systems have a unique failure taxonomy. Retrieval can fail (wrong chunks, not enough chunks, irrelevant chunks). Generation can fail (hallucinating beyond the context, ignoring relevant context, blending facts from multiple sources). Standard generation metrics cannot distinguish these failure modes. You need RAG-specific evaluation.
Why RAG Evaluation Is Different
Standard LLM evaluation asks: "Is the output good?" RAG evaluation must ask three separate questions:
- Is the output grounded in what was retrieved? (Faithfulness)
- Did we retrieve what was needed? (Context relevance)
- Does the output answer the question? (Answer relevance)
These are independent failure modes. A system can:
- Retrieve perfectly relevant context but hallucinate beyond it (faithfulness failure)
- Produce a faithful response to what was retrieved but retrieve irrelevant content (context failure)
- Retrieve and faithfully reproduce context that doesn't answer the question (relevance failure)
Standard metrics conflate these. If your answer ROUGE score drops, you do not know if the retrieval got worse or the generation got worse. Component-wise evaluation is the only way to diagnose and fix RAG failures.
The RAG Triad (TruLens Framework)
TruLens introduced the RAG Triad as a structured framework for evaluating the three core RAG failure modes:
Faithfulness (Groundedness)
Definition: Does every claim in the generated answer appear in or follow from the retrieved context?
Computation:
- Extract all atomic claims from the generated answer
- For each claim, check whether it is supported by the retrieved context
- Faithfulness = (supported claims) / (total claims)
A faithfulness score of 1.0 means the model never makes claims beyond what is in the retrieved context. A score of 0.5 means half the claims are hallucinated.
Context Relevance
Definition: Are the retrieved chunks actually relevant to the user's question?
Computation: For each retrieved chunk, assess whether it contains information that is relevant and necessary for answering the question. Context relevance = (relevant chunks) / (total retrieved chunks)
Low context relevance means the retriever is returning noisy results - content that is topically related but doesn't answer the specific question.
Answer Relevance
Definition: Does the generated answer actually address the user's question?
Computation: Generate questions from the answer and measure the semantic similarity between those questions and the original question. High similarity means the answer is focused on the question.
This catches a common failure mode: the model produces a correct, faithful response to the retrieved context that never actually answers the user's question (because the context was off-topic).
RAGAS Framework
RAGAS (Retrieval-Augmented Generation Assessment, Es et al., 2023) is the most widely used framework for automated RAG evaluation. It implements the RAG triad plus additional metrics and provides a Python library for easy integration.
RAGAS Metrics
Faithfulness: As described above - atomic claim extraction + verification against context.
Answer Relevancy: Uses a reverse generation approach:
- Generate multiple questions from the answer (what question would this answer?)
- Compute cosine similarity between generated questions and the original question
- Higher similarity = more relevant answer
Context Precision: Are the relevant chunks ranked higher than irrelevant ones?
where is 1 if the k-th chunk is relevant, 0 otherwise.
Context Recall: Does the retrieved context contain all the information needed to answer the question?
Context recall requires a ground truth answer to compute. It checks whether each statement in the ground truth can be attributed to the retrieved context.
Answer Correctness: Semantic similarity between the generated answer and the ground truth answer, combining factual overlap and semantic similarity.
Running RAGAS
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
answer_correctness,
)
from datasets import Dataset
from typing import List, Optional
import pandas as pd
def run_ragas_evaluation(
questions: List[str],
answers: List[str],
contexts: List[List[str]],
ground_truths: Optional[List[str]] = None,
) -> dict:
"""
Run a complete RAGAS evaluation.
Args:
questions: User questions
answers: Generated answers from your RAG system
contexts: Retrieved context for each question (list of strings per question)
ground_truths: Reference answers (required for context_recall and answer_correctness)
Returns:
RAGAS evaluation scores
"""
# Build dataset
data = {
"question": questions,
"answer": answers,
"contexts": contexts,
}
if ground_truths is not None:
data["ground_truth"] = ground_truths
dataset = Dataset.from_dict(data)
# Select metrics based on available ground truth
if ground_truths is not None:
metrics = [
faithfulness,
answer_relevancy,
context_precision,
context_recall,
answer_correctness,
]
else:
# Without ground truth, can only compute faithfulness and answer_relevancy
metrics = [faithfulness, answer_relevancy]
print("Running RAGAS evaluation...")
results = evaluate(dataset, metrics=metrics)
print("\n=== RAGAS Scores ===")
for metric_name, score in results.items():
print(f"{metric_name:<25} {score:.4f}")
return results
# Example: evaluating a RAG system
def evaluate_rag_system_example():
"""
End-to-end example of RAGAS evaluation.
Replace model_fn and retriever_fn with your actual implementation.
"""
# Sample test data
test_questions = [
"What is the main cause of the French Revolution?",
"How does photosynthesis work?",
"What were the main provisions of the Treaty of Versailles?",
]
# Simulate retrieval results (in production, these come from your vector database)
retrieved_contexts = [
[
"The French Revolution (1789-1799) was caused by financial crisis, food shortages, "
"and growing resentment toward the privileges of the nobility and clergy.",
"France's involvement in the American Revolution had left the country deeply in debt.",
],
[
"Photosynthesis is the process by which plants convert sunlight, water, and carbon "
"dioxide into glucose and oxygen using chlorophyll in chloroplasts.",
"The light-dependent reactions occur in the thylakoid membranes, while the "
"Calvin cycle occurs in the stroma.",
],
[
"The Treaty of Versailles (1919) required Germany to accept full responsibility "
"for World War I and imposed heavy reparations of 132 billion gold marks.",
"Germany was required to significantly reduce its military forces and cede "
"territory to neighboring countries.",
],
]
# Simulate generated answers (in production, these come from your LLM)
generated_answers = [
"The French Revolution was primarily caused by France's severe financial crisis, "
"food shortages among the poor, and resentment of aristocratic privileges. "
"Debt from the American Revolution also contributed significantly.",
"Photosynthesis is the process where plants use sunlight, water, and CO2 to "
"produce glucose and oxygen. This occurs in chloroplasts using chlorophyll.",
"The Treaty of Versailles imposed the war guilt clause on Germany, required "
"reparations of 132 billion gold marks, limited Germany's military, and "
"required territorial concessions.",
]
# Ground truth answers (for context recall and answer correctness)
ground_truths = [
"The main causes were financial crisis, food shortages, and resentment of noble privileges.",
"Photosynthesis converts sunlight, CO2, and water into glucose and oxygen via chloroplasts.",
"The Treaty imposed war guilt, reparations, military limits, and territorial changes on Germany.",
]
results = run_ragas_evaluation(
questions=test_questions,
answers=generated_answers,
contexts=retrieved_contexts,
ground_truths=ground_truths,
)
return results
Building a Golden Dataset
A golden dataset is a curated set of question-answer pairs with known correct answers, used for systematic evaluation. It is the cornerstone of any serious RAG evaluation pipeline.
Manual Golden Dataset Creation
from anthropic import Anthropic
from typing import List
import json
import random
def generate_golden_dataset(
source_documents: List[str],
n_questions: int = 100,
question_types: List[str] = None,
) -> List[dict]:
"""
Generate a golden evaluation dataset from source documents.
Uses Claude to generate diverse question types.
Args:
source_documents: List of document strings to generate questions from
n_questions: Total number of questions to generate
question_types: Types of questions to include
Returns:
List of {question, context, ground_truth, question_type, difficulty}
"""
if question_types is None:
question_types = [
"factual", # Direct fact retrieval
"reasoning", # Requires combining multiple facts
"multi_hop", # Answer requires following a chain of references
"adversarial", # Designed to test robustness (e.g., similar but wrong phrasing)
"open_ended", # No single right answer but must be grounded
]
client = Anthropic()
golden_set = []
n_per_type = n_questions // len(question_types)
for q_type in question_types:
for _ in range(n_per_type):
# Randomly select a document chunk
doc = random.choice(source_documents)
doc_chunk = doc[:2000] # Limit for API
prompt = f"""Based on the following document, generate a {q_type} question and its ideal answer.
Document:
{doc_chunk}
Question type: {q_type}
- factual: Ask about a specific fact directly stated in the document
- reasoning: Ask something that requires combining 2-3 facts from the document
- multi_hop: Ask something where the answer requires following references
- adversarial: Use similar but slightly wrong phrasing that might confuse a retrieval system
- open_ended: Ask for analysis or summary that has multiple valid answers
Generate:
1. A specific question
2. The ideal ground truth answer (based only on this document)
3. Difficulty (easy/medium/hard)
Format as JSON:
{{"question": "...", "ground_truth": "...", "difficulty": "easy/medium/hard"}}"""
response = client.messages.create(
model="claude-3-5-haiku-20241022",
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
try:
qa_pair = json.loads(response.content[0].text)
qa_pair["question_type"] = q_type
qa_pair["source_document"] = doc_chunk[:200]
golden_set.append(qa_pair)
except (json.JSONDecodeError, KeyError):
continue
return golden_set
def validate_golden_dataset(
golden_set: List[dict],
rag_system_fn, # Function: question -> (answer, contexts)
) -> dict:
"""
Validate the golden dataset by checking if it is solvable.
A good golden dataset should be answerable by the RAG system
when given the correct context.
"""
solvable = 0
for item in golden_set[:20]: # Sample validation
answer, contexts = rag_system_fn(item["question"])
# Check if ground truth info is in retrieved contexts
gt_words = set(item["ground_truth"].lower().split())
context_words = set(" ".join(contexts).lower().split())
overlap = len(gt_words.intersection(context_words)) / len(gt_words)
if overlap > 0.5: # 50% word overlap threshold
solvable += 1
return {
"sample_solvability_rate": round(solvable / min(20, len(golden_set)), 4),
"total_questions": len(golden_set),
}
Retrieval Metrics
Component evaluation should isolate retrieval from generation. The following metrics evaluate the retrieval component independently.
MRR (Mean Reciprocal Rank)
For each query, what is the rank of the first relevant document?
If the first relevant document is at rank 1, reciprocal rank = 1. If at rank 3, = 1/3. MRR penalizes systems that bury the most relevant document.
NDCG@K (Normalized Discounted Cumulative Gain)
More nuanced than MRR - accounts for graded relevance (highly relevant vs somewhat relevant) and position:
where IDCG is the ideal DCG (perfect ranking). NDCG@K ranges from 0 to 1.
Recall@K
What fraction of all relevant documents appear in the top K retrieved?
For RAG, Recall@K is critical because missing relevant documents means the generation step cannot possibly produce a complete answer.
from typing import List, Set
import numpy as np
def compute_retrieval_metrics(
queries: List[str],
retrieved_doc_ids: List[List[str]], # Top-K retrieved for each query
relevant_doc_ids: List[Set[str]], # Ground truth relevant docs per query
k_values: List[int] = [1, 3, 5, 10],
) -> dict:
"""
Compute retrieval quality metrics: MRR, NDCG@K, Recall@K, Precision@K.
Args:
queries: List of query strings
retrieved_doc_ids: For each query, ordered list of retrieved document IDs
relevant_doc_ids: For each query, set of ground-truth relevant document IDs
k_values: Values of K to compute metrics at
"""
n_queries = len(queries)
mrr_scores = []
ndcg_scores = {k: [] for k in k_values}
recall_scores = {k: [] for k in k_values}
precision_scores = {k: [] for k in k_values}
for retrieved, relevant in zip(retrieved_doc_ids, relevant_doc_ids):
# MRR: reciprocal rank of first relevant document
rr = 0.0
for rank, doc_id in enumerate(retrieved, start=1):
if doc_id in relevant:
rr = 1.0 / rank
break
mrr_scores.append(rr)
for k in k_values:
top_k = retrieved[:k]
# Recall@K
recall = len(set(top_k) & relevant) / len(relevant) if relevant else 0
recall_scores[k].append(recall)
# Precision@K
precision = len(set(top_k) & relevant) / k if k > 0 else 0
precision_scores[k].append(precision)
# NDCG@K (binary relevance)
dcg = sum(
1.0 / np.log2(rank + 2) # rank 0-indexed, so +2 for log_2(i+1) formula
for rank, doc_id in enumerate(top_k)
if doc_id in relevant
)
# Ideal DCG: all relevant docs at top positions
n_ideal = min(len(relevant), k)
idcg = sum(1.0 / np.log2(rank + 2) for rank in range(n_ideal))
ndcg = dcg / idcg if idcg > 0 else 0
ndcg_scores[k].append(ndcg)
results = {
"mrr": round(np.mean(mrr_scores), 4),
"n_queries": n_queries,
}
for k in k_values:
results[f"ndcg@{k}"] = round(np.mean(ndcg_scores[k]), 4)
results[f"recall@{k}"] = round(np.mean(recall_scores[k]), 4)
results[f"precision@{k}"] = round(np.mean(precision_scores[k]), 4)
return results
# Example
def example_retrieval_eval():
queries = ["What is the capital of France?", "Who wrote Hamlet?"]
retrieved = [
["doc_paris", "doc_france_1", "doc_europe", "doc_london", "doc_germany"],
["doc_shakespeare_1", "doc_hamlet_plot", "doc_shakespeare_2", "doc_othello"],
]
relevant = [
{"doc_paris", "doc_france_1"},
{"doc_shakespeare_1", "doc_hamlet_plot"},
]
metrics = compute_retrieval_metrics(queries, retrieved, relevant)
for metric, value in sorted(metrics.items()):
print(f"{metric:<20} {value}")
Component Ablation
To diagnose RAG failures, you need to isolate each component. The standard ablation protocol:
def run_component_ablation(
questions: List[str],
ground_truths: List[str],
rag_system,
oracle_retriever, # Returns perfect ground-truth context
standalone_llm, # LLM without retrieval (direct QA)
) -> dict:
"""
Ablation study to isolate RAG component contributions.
Conditions:
1. Full RAG (production): retriever + generator
2. Oracle retrieval: perfect context + generator (measures generation ceiling)
3. No retrieval: LLM alone without context (measures base LLM knowledge)
"""
results = {
"full_rag": [],
"oracle_retrieval": [],
"no_retrieval": [],
}
for question, gt in zip(questions, ground_truths):
# Condition 1: Full RAG
full_answer, full_contexts = rag_system(question)
# Condition 2: Oracle retrieval (give the model perfect context)
oracle_contexts = oracle_retriever(question)
oracle_answer = rag_system.generate_only(question, oracle_contexts)
# Condition 3: No retrieval
no_retrieval_answer = standalone_llm(question)
results["full_rag"].append({
"question": question,
"answer": full_answer,
"ground_truth": gt,
})
results["oracle_retrieval"].append({
"question": question,
"answer": oracle_answer,
"ground_truth": gt,
})
results["no_retrieval"].append({
"question": question,
"answer": no_retrieval_answer,
"ground_truth": gt,
})
# Evaluate each condition with RAGAS
ablation_scores = {}
for condition, condition_results in results.items():
questions_list = [r["question"] for r in condition_results]
answers_list = [r["answer"] for r in condition_results]
gts_list = [r["ground_truth"] for r in condition_results]
# Note: for no_retrieval, pass empty contexts
contexts_list = [["No retrieval context"] for _ in questions_list]
ablation_scores[condition] = run_ragas_evaluation(
questions=questions_list,
answers=answers_list,
contexts=contexts_list,
ground_truths=gts_list,
)
print("\n=== Component Ablation Results ===")
for condition, scores in ablation_scores.items():
print(f"\n{condition}:")
for metric, score in scores.items():
print(f" {metric:<25} {score:.4f}")
# Interpretation guide
print("\n=== Interpretation ===")
full_faith = ablation_scores["full_rag"].get("faithfulness", 0)
oracle_faith = ablation_scores["oracle_retrieval"].get("faithfulness", 0)
no_ret_acc = ablation_scores["no_retrieval"].get("answer_correctness", 0)
if full_faith < oracle_faith - 0.1:
print("Generation issue: Model hallucinates less with perfect context → "
"generation is the bottleneck, not retrieval.")
if oracle_faith > 0.9 and full_faith < 0.7:
print("Retrieval issue: Generator performs well with oracle context but "
"poorly with actual retrieval → improve retrieval quality.")
if no_ret_acc > 0.7:
print("Knowledge issue: Model answers well without retrieval → "
"RAG may not be needed for this query type.")
return ablation_scores
Production: Continuous Evaluation
import asyncio
from datetime import datetime, timedelta
from collections import defaultdict
class RAGProductionMonitor:
"""
Continuous evaluation for production RAG systems.
Tracks faithfulness, context quality, and answer relevance over time.
"""
def __init__(
self,
golden_dataset: List[dict],
eval_interval_hours: int = 24,
sample_rate: float = 0.05, # Evaluate 5% of production traffic
):
self.golden_dataset = golden_dataset
self.eval_interval_hours = eval_interval_hours
self.sample_rate = sample_rate
self.metrics_history = defaultdict(list)
def should_sample_request(self) -> bool:
"""Reservoir sampling decision for production traffic."""
import random
return random.random() < self.sample_rate
def log_request_for_eval(
self,
question: str,
answer: str,
contexts: List[str],
timestamp: datetime = None,
) -> None:
"""Log a sampled production request for async evaluation."""
if timestamp is None:
timestamp = datetime.utcnow()
self.metrics_history["pending"].append({
"question": question,
"answer": answer,
"contexts": contexts,
"timestamp": timestamp.isoformat(),
})
def run_scheduled_evaluation(self) -> dict:
"""
Run scheduled evaluation on golden dataset + sampled production traffic.
Should be triggered on a schedule (daily recommended).
"""
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset
# 1. Golden dataset evaluation
golden_questions = [g["question"] for g in self.golden_dataset[:50]]
golden_gts = [g["ground_truth"] for g in self.golden_dataset[:50]]
# These would come from actually running your RAG system
# (simplified here - in production, run your live system)
golden_answers = golden_questions # Placeholder
golden_contexts = [["context"] for _ in golden_questions] # Placeholder
golden_data = Dataset.from_dict({
"question": golden_questions,
"answer": golden_answers,
"contexts": golden_contexts,
"ground_truth": golden_gts,
})
golden_scores = evaluate(
golden_data,
metrics=[faithfulness, answer_relevancy, context_precision],
)
# 2. Production sample evaluation (no ground truth available)
pending = self.metrics_history.get("pending", [])
production_scores = {}
if len(pending) >= 10:
prod_data = Dataset.from_dict({
"question": [p["question"] for p in pending[:100]],
"answer": [p["answer"] for p in pending[:100]],
"contexts": [p["contexts"] for p in pending[:100]],
})
production_scores = evaluate(
prod_data,
metrics=[faithfulness, answer_relevancy],
)
# 3. Drift detection
current_time = datetime.utcnow().isoformat()
self.metrics_history["golden_eval"].append({
"timestamp": current_time,
"scores": dict(golden_scores),
})
# Alert on quality degradation
alerts = self._check_for_drift()
# Clear pending queue
self.metrics_history["pending"] = []
return {
"golden_scores": dict(golden_scores),
"production_scores": dict(production_scores),
"alerts": alerts,
"evaluated_at": current_time,
}
def _check_for_drift(self) -> List[str]:
"""Check for quality degradation by comparing recent evals."""
history = self.metrics_history.get("golden_eval", [])
alerts = []
if len(history) < 3:
return alerts
# Compare most recent eval to 7-day average
recent = history[-1]["scores"]
week_ago = history[max(0, len(history)-7):-1]
for metric in ["faithfulness", "answer_relevancy"]:
recent_score = recent.get(metric, 0)
if week_ago:
historical_avg = sum(
h["scores"].get(metric, 0) for h in week_ago
) / len(week_ago)
if recent_score < historical_avg - 0.05: # 5% degradation
alerts.append(
f"ALERT: {metric} degraded from {historical_avg:.3f} "
f"to {recent_score:.3f} (delta: {recent_score - historical_avg:+.3f})"
)
return alerts
:::tip Use RAGAS Without Ground Truth First You do not need a golden dataset to start evaluating. Faithfulness and answer relevancy can be computed without ground truth answers. Start by evaluating these two metrics on production samples. Add context precision and recall later when you have a labeled dataset. :::
:::warning RAGAS Metrics Are LLM-Based RAGAS uses LLM calls internally to compute metrics - including extracting atomic claims for faithfulness. This means RAGAS results can vary across runs (non-deterministic), have their own LLM biases, and cost API money. For production monitoring, compute RAGAS metrics on a sample, not every request. :::
Common Mistakes
:::danger Evaluating RAG as a Single System If RAGAS scores drop, you do not know if retrieval got worse, generation got worse, or your source documents changed. Always maintain separate retrieval metrics (MRR, Recall@K) and generation metrics (faithfulness, answer relevancy) so you can diagnose where failures originate. :::
:::warning Trusting LLM-Generated Golden Datasets Without Validation Using Claude or GPT-4 to generate golden Q&A pairs is efficient, but the generated questions may be biased toward patterns the LLM is good at answering. Always validate a sample of your golden dataset with domain experts, especially for specialized applications. :::
:::danger Ignoring Context Window Limitations RAGAS faithfulness computation works by passing the entire context to a judge LLM. For RAG systems with large retrieved context (10+ documents × 500 tokens each), this can exceed context limits or produce unreliable evaluations because the judge cannot carefully read 5,000 tokens of context. Chunk your evaluation into smaller pieces or use a model with a large context window. :::
Interview Q&A
Q1: What is the RAG triad and why does each component need separate evaluation?
The RAG triad (from TruLens) consists of: (1) Faithfulness - does the answer stay grounded in the retrieved context without hallucinating? (2) Context relevance - did the retriever return chunks that are relevant to the question? (3) Answer relevance - does the generated answer actually address the user's question? Each needs separate evaluation because they represent independent failure modes. A system can fail faithfulness while having perfect retrieval (good context, but the generator hallucinates). It can fail context relevance while having perfect generation (the generator faithfully reproduces off-topic context). It can fail answer relevance while having both perfect retrieval and faithful generation (retrieved content is real and faithfully reproduced, but doesn't answer the question). Aggregate metrics like ROUGE cannot distinguish these cases.
Q2: How does RAGAS compute faithfulness?
RAGAS faithfulness works in two steps: (1) Claim extraction - it passes the generated answer to an LLM and asks it to extract all atomic, verifiable claims. For example, the answer "The Treaty of Versailles was signed in 1919 and required Germany to pay 132 billion marks in reparations" would yield two claims: "signed in 1919" and "required Germany to pay 132 billion marks." (2) Claim verification - for each extracted claim, it passes both the claim and the retrieved context to an LLM and asks whether the claim is supported by the context. Faithfulness = supported claims / total claims. This is LLM-based, so it has non-determinism and can miss subtle hallucinations, but it is significantly more reliable than lexical or embedding-based similarity metrics for detecting out-of-context claims.
Q3: What is the difference between context precision and context recall in RAGAS?
Context precision and context recall evaluate the retriever from different angles. Context precision asks: of the chunks that were retrieved, what fraction are actually relevant? High precision means the retriever avoids noise. Context recall asks: of all the information needed to answer the question correctly, what fraction is present in the retrieved chunks? High recall means the retriever does not miss important information. Both matter but have different failure modes. Low precision means the generator is working with noisy context, which may cause confusion or hallucination. Low recall means even a perfect generator cannot produce a complete answer because the information it needs was never retrieved.
Q4: How would you build a golden dataset for a RAG system that answers questions about company policies?
Start by identifying the full range of question types: simple factual (What is the vacation policy?), comparative (How does the PTO policy differ for full-time vs part-time?), multi-hop (What happens if I exceed my expense limit and my manager is unavailable?), and edge cases (Questions that sound like they're about policies but actually require interpretation). For each type, manually write 20–30 representative examples. For the factual and comparative questions, the ground truth answer is the relevant policy text. For multi-hop and interpretation questions, have HR experts write the ground truth. Include "unanswerable" questions where the policy doesn't address the case (these test whether the system says "I don't know" instead of hallucinating). Validate the dataset by running your system on it and checking whether context recall is above 0.8 (the context is actually there). Update the golden set quarterly when policies change.
Q5: A RAG system's faithfulness score drops from 0.85 to 0.65 after a software update. How do you diagnose?
Start with component isolation: (1) Check if retrieval changed - run the same queries and compare retrieved contexts before and after. If contexts changed, retrieval is contributing to the drop. (2) Check if generation changed - run the same queries with the same manually-verified perfect contexts (oracle retrieval). If faithfulness drops even with oracle context, the generator changed (e.g., different model, changed system prompt, different temperature). (3) Check if the documents changed - if the knowledge base was updated, some previously faithful answers may now contradict updated facts. (4) Look at the claim-level failures - RAGAS tells you which claims are not supported. Cluster them by failure type: are they specific facts the model is inventing, or general hallucinations? (5) Check the prompt - if the system prompt changed, the model may be generating in a style that adds confident-sounding claims not in the context. The 0.20 drop in faithfulness suggests a significant change - likely a system prompt or model update that changed generation behavior, or a large batch of knowledge base updates.
:::tip 🎮 Interactive Playground
Visualize this concept: Try the RAG Pipeline demo on the EngineersOfAI Playground - no code required.
:::