:::tip 🎮 Interactive Playground Visualize this concept: Try the RAG Evaluation demo on the EngineersOfAI Playground - no code required. :::
RAG-Specific Evaluation
Reading time: 38 minutes | Interview relevance: Very High | Target roles: AI Engineer, ML Engineer, RAG Systems Engineer, MLOps
The Invisible Retrieval Failure
The company's documentation assistant had passed every quality gate with strong scores. Helpfulness: 4.2/5. Tone appropriateness: 4.6/5. Formatting quality: 4.7/5. Response completeness: 4.1/5. The evaluation dashboard was green across the board, and the team had shipped with confidence after three rounds of testing.
Then the support tickets started arriving. Not a flood - a trickle. But each one described the same pattern: the AI had confidently told users to use a feature that was deprecated in v2.0, referencing an API that had been removed eight months ago. The documentation the assistant was retrieving was correct at time of ingestion, but the product had moved on. The AI was faithfully answering from its retrieved context - the problem was that the retrieved context was wrong, outdated, or entirely superseded by newer documentation that should have ranked higher in retrieval.
The general evaluation framework had completely missed this because it was testing generation quality, not retrieval accuracy or faithfulness to the specific retrieved context. A response could score 4.5/5 on helpfulness while being entirely wrong about the actual product. Helpfulness measures whether the response feels complete and useful to a reader who does not know the correct answer. It does not measure whether the response is grounded in the retrieved documents, whether the retrieved documents were the right ones, or whether outdated information was ranked above current information.
This is the fundamental gap in applying general LLM evaluation to RAG systems. RAG has two independent failure modes: retrieval failure and generation failure. A response can fail because the right documents were never retrieved (retrieval failure) - the generation could be perfectly faithful to what it was given, but what it was given was wrong. Or a response can fail because the right documents were retrieved but the model ignored them in favor of parametric knowledge baked in during pre-training (generation failure). A general helpfulness score collapses both failure modes into a single number, making them impossible to distinguish and diagnose.
The team rebuilt their evaluation stack from scratch with RAG-specific metrics. They implemented faithfulness scoring to detect when the model answered from parametric memory rather than from retrieved documents. They implemented context precision to score whether retrieved chunks were actually relevant. They added temporal faithfulness checking to detect when the model used an older retrieved document when a newer one was also available. Within two weeks of deployment, they had identified and fixed three distinct retrieval bugs that the original evaluation had completely missed.
Why RAG Evaluation Is Different
RAG evaluation is not a subset of LLM evaluation - it is a separate discipline because the failure taxonomy is fundamentally different.
Two independent failure modes. A traditional LLM either produces a good response or it does not. A RAG system can fail at retrieval (wrong documents), fail at generation (ignores correct documents), or fail at both simultaneously. These failures have different root causes and different fixes: retrieval failures require changes to the retriever, generation failures require changes to the prompt or model. If you cannot distinguish them, you cannot diagnose them.
Faithfulness as a first-class metric. In a standard LLM, "factual accuracy" means comparing the response to world knowledge. In a RAG system, "faithfulness" means something narrower and more measurable: does the response make only claims that are supported by the specific documents that were retrieved? A response can be factually true but unfaithful to the context - meaning the model answered from its parametric weights rather than from what it was given. This is a critical failure mode in high-stakes RAG applications where the retrieved documents are authoritative (legal, medical, technical documentation).
Attribution and citation accuracy. Many RAG systems present citations alongside answers. Evaluation must check whether each cited source actually supports the claim it is cited for. This is a different check from faithfulness - a response can be faithful to the aggregate context while still misattributing which specific chunk supported which specific claim.
Compound errors. Bad retrieval leads to bad generation, but both must be evaluated separately. If you only measure final answer quality, bad retrieval masked by a plausible-sounding hallucination will score the same as a system that retrieved correctly and generated accurately. Compound evaluation - measuring each stage independently - is required to identify which stage is the bottleneck.
Historical Context
The RAGAS framework (RAG Assessment) was introduced in 2023 by Shahul Es and colleagues as the first systematic framework for evaluating RAG systems without requiring human-annotated reference answers for every test case. RAGAS introduced the observation that faithfulness and answer relevance could be computed using the LLM itself as a judge, making large-scale evaluation practical.
Prior to RAGAS, RAG evaluation was done primarily through human evaluation of final responses, which was expensive and could not decompose retrieval from generation failures. Information retrieval evaluation (precision@k, recall@k, nDCG) had existed since the 1960s in the IR community, but these metrics measure retrieval quality alone and ignore generation quality entirely.
The challenge RAGAS addressed - evaluating faithfulness without a ground-truth answer for comparison - is solved by asking the question in reverse: instead of "does the answer match a reference?", ask "are the claims in the answer supported by the retrieved context?" This makes faithfulness evaluation tractable at scale using an LLM judge.
The Four Core RAGAS Metrics
1. Faithfulness
Faithfulness measures whether the claims in the generated answer are supported by the retrieved context. It is defined as:
Implementation requires two steps: (1) extract atomic claims from the answer, (2) verify each claim against the retrieved context. A claim is faithful if the context explicitly states or logically implies it.
2. Answer Relevance
Answer relevance measures whether the answer addresses the actual question. It is computed by generating multiple hypothetical questions from the answer and measuring their similarity to the original question. If the answer truly addresses the question, questions generated from it should be similar to the original:
where are reverse-generated questions from the answer.
3. Context Precision
Context precision measures what fraction of the retrieved chunks are actually relevant to the question. It penalizes retrieving many irrelevant documents alongside the relevant ones:
4. Context Recall
Context recall measures what fraction of the ground-truth statements are covered by the retrieved context. It requires a reference answer and checks whether the retrieved documents contain the information needed to produce it:
Beyond RAGAS - Additional RAG Metrics
Hallucination Type Classification
Hallucinations in RAG systems are not monolithic. There are three distinct types with different diagnostic implications:
Out-of-context hallucination. The model uses parametric knowledge (information from pre-training) that is not present in the retrieved context. This is a generation failure: the retriever may have provided relevant documents, but the model ignored them in favor of what it already knew. Fix: stronger prompt instructions to use only retrieved content, or a model fine-tuned for RAG faithfulness.
Inconsistency hallucination. The model produces a claim that directly contradicts something in the retrieved context. This is the most severe type - the model is not just ignoring context, it is generating content that conflicts with what it was given. Fix: usually a model quality issue, or the retrieved context itself contains contradictions (e.g., documents from different time periods with conflicting information).
Fabrication hallucination. The model invents citations, document names, or sources that do not exist in the retrieved context. Fix: strict output format enforcement and post-processing citation verification.
Citation Accuracy
Citation accuracy measures whether each citation in a response actually supports the claim it is cited for. A system can achieve high faithfulness (all claims are somewhere in the retrieved context) while having poor citation accuracy (claims are attributed to the wrong chunks). For high-stakes applications where users need to verify sources, citation accuracy is critical.
Temporal Faithfulness
When multiple retrieved documents cover the same topic but from different time periods, temporal faithfulness measures whether the model correctly uses the most recent document. A system with poor temporal faithfulness will answer using an older v1.0 documentation chunk when a v2.0 chunk was also retrieved.
Retrieval Diversity
Retrieval diversity measures whether the retrieved chunks provide diverse, complementary information rather than redundant coverage of the same content. A retriever that returns five nearly identical chunks wastes context window and provides no more information than a single chunk would.
Retrieval-Specific Evaluation Metrics
Precision@k
Measures what fraction of retrieved chunks are actually relevant. High precision means few irrelevant chunks in context - important because irrelevant context can confuse the generation model and waste tokens.
Recall@k
Measures what fraction of all relevant chunks were retrieved. High recall ensures the generation model has access to all the information it needs.
nDCG@k (Normalized Discounted Cumulative Gain)
nDCG@k measures not just whether relevant chunks are retrieved but whether they are ranked first. Chunks ranked higher in the retrieval result receive higher weight:
where IDCG@k is the ideal (maximum possible) DCG for that query.
Mean Reciprocal Rank (MRR)
MRR is the right metric when you care specifically about whether the first relevant chunk appears near the top of results - critical for single-passage extraction tasks.
Hit Rate@k
Hit rate@k is the simplest useful metric: for what fraction of queries does at least one relevant chunk appear in the top-k results?
Complete Implementation
import anthropic
import asyncio
import statistics
import math
import re
from dataclasses import dataclass, field
from typing import Optional
from enum import Enum
from collections import defaultdict
client = anthropic.Anthropic()
async_client = anthropic.AsyncAnthropic()
# ---------------------------------------------------------------------------
# Data Structures
# ---------------------------------------------------------------------------
class HallucinationType(Enum):
GROUNDED = "grounded" # Supported by context
OUT_OF_CONTEXT = "out_of_context" # From parametric memory
INCONSISTENCY = "inconsistency" # Contradicts context
FABRICATION = "fabrication" # Invented citation/source
@dataclass
class RetrievedChunk:
chunk_id: str
content: str
source_doc: str
doc_version: Optional[str] = None
timestamp: Optional[str] = None
relevance_score: float = 0.0 # Score from retriever
is_relevant: Optional[bool] = None # Ground truth label
@dataclass
class RAGEvalDatapoint:
question: str
answer: str
retrieved_chunks: list[RetrievedChunk]
ground_truth: Optional[str] = None
citations: Optional[dict[str, str]] = None # claim -> chunk_id
metadata: dict = field(default_factory=dict)
@dataclass
class RetrievalMetrics:
precision_at_k: float
recall_at_k: float
ndcg_at_k: float
mrr: float
hit_rate: float
k: int
@dataclass
class ClaimVerification:
claim: str
is_faithful: bool
supporting_chunk_id: Optional[str]
confidence: float
hallucination_type: HallucinationType
@dataclass
class CitationReport:
total_citations: int
accurate_citations: int
citation_accuracy: float
inaccurate: list[dict] # {claim, cited_chunk_id, reason}
@dataclass
class RAGEvalReport:
faithfulness: float
answer_relevance: float
context_precision: float
context_recall: float
hallucination_breakdown: dict[str, float]
citation_accuracy: Optional[float]
retrieval_metrics: Optional[RetrievalMetrics]
identified_issues: list[str]
overall_score: float
@dataclass
class WeakPoint:
metric: str
score: float
description: str
recommended_fix: str
@dataclass
class ComparisonReport:
system_a_scores: dict[str, float]
system_b_scores: dict[str, float]
improvements: dict[str, float] # metric -> delta (positive = B better)
regressions: dict[str, float] # metric -> delta (negative = B worse)
recommendation: str
# ---------------------------------------------------------------------------
# Hallucination Classifier
# ---------------------------------------------------------------------------
class HallucinationClassifier:
"""
Classifies individual claims in a RAG response into hallucination types.
Uses claude-haiku for cost-effective large-scale classification.
"""
CLASSIFICATION_PROMPT = """You are evaluating whether a claim from an AI response is supported by retrieved context.
Question: {question}
Retrieved context:
{context}
Claim to evaluate: {claim}
Classify this claim into exactly one category:
- GROUNDED: The context explicitly states or clearly implies this claim
- OUT_OF_CONTEXT: This claim may be true generally but is not in the retrieved context (model used parametric memory)
- INCONSISTENCY: This claim directly contradicts something in the retrieved context
- FABRICATION: This claim invents a source, citation, or document that does not appear in context
Respond with JSON: {{"type": "GROUNDED|OUT_OF_CONTEXT|INCONSISTENCY|FABRICATION", "confidence": 0.0-1.0, "evidence": "brief explanation", "supporting_chunk_id": "chunk_id or null"}}"""
def classify_claim(
self,
claim: str,
question: str,
retrieved_chunks: list[RetrievedChunk],
) -> ClaimVerification:
"""Classify a single claim against the retrieved context."""
context_text = "\n\n".join(
f"[{chunk.chunk_id}] {chunk.content}"
for chunk in retrieved_chunks
)
prompt = self.CLASSIFICATION_PROMPT.format(
question=question,
context=context_text[:4000],
claim=claim,
)
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": prompt}],
)
raw = response.content[0].text.strip()
try:
import json
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
h_type = HallucinationType[result["type"]]
confidence = float(result.get("confidence", 0.8))
supporting_chunk_id = result.get("supporting_chunk_id")
except Exception:
h_type = HallucinationType.OUT_OF_CONTEXT
confidence = 0.5
supporting_chunk_id = None
return ClaimVerification(
claim=claim,
is_faithful=h_type == HallucinationType.GROUNDED,
supporting_chunk_id=supporting_chunk_id,
confidence=confidence,
hallucination_type=h_type,
)
def extract_claims(
self, answer: str, question: str
) -> list[str]:
"""
Extract atomic, verifiable claims from an answer.
Each claim should be a single, independently verifiable statement.
"""
prompt = f"""Break this AI response into a list of atomic, independently verifiable claims.
Question: {question}
Response: {answer}
Rules:
- Each claim must be a single factual statement
- Do not include opinions or subjective statements
- Do not include "According to..." framing - just the factual claim itself
- Exclude claims that are trivially true (e.g., "This is a complex topic")
Output one claim per line, no numbering:"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
claims = [
line.strip()
for line in response.content[0].text.strip().split("\n")
if line.strip() and len(line.strip()) > 10
]
return claims
def classify_all_claims(
self,
answer: str,
question: str,
retrieved_chunks: list[RetrievedChunk],
) -> list[ClaimVerification]:
"""Extract and classify all claims in an answer."""
claims = self.extract_claims(answer, question)
verifications = []
for claim in claims:
verification = self.classify_claim(claim, question, retrieved_chunks)
verifications.append(verification)
return verifications
def compute_hallucination_rates(
self,
verifications: list[ClaimVerification],
) -> dict[str, float]:
"""Compute fraction of claims in each hallucination category."""
if not verifications:
return {h.value: 0.0 for h in HallucinationType}
n = len(verifications)
counts: dict[str, int] = defaultdict(int)
for v in verifications:
counts[v.hallucination_type.value] += 1
return {h_type: count / n for h_type, count in counts.items()}
# ---------------------------------------------------------------------------
# Citation Accuracy Evaluator
# ---------------------------------------------------------------------------
class CitationAccuracyEvaluator:
"""
Verifies whether cited sources actually support the claims they are cited for.
Critical for RAG systems that present sources alongside answers.
"""
VERIFICATION_PROMPT = """Does the following chunk of text support the given claim?
Claim: {claim}
Text chunk [{chunk_id}]:
{chunk_content}
Answer with JSON: {{"supports": true/false, "confidence": 0.0-1.0, "reason": "brief explanation"}}"""
def verify_citation(
self,
claim: str,
cited_chunk: RetrievedChunk,
) -> tuple[bool, float, str]:
"""
Check whether cited_chunk actually supports claim.
Returns (supports, confidence, reason).
"""
prompt = self.VERIFICATION_PROMPT.format(
claim=claim,
chunk_id=cited_chunk.chunk_id,
chunk_content=cited_chunk.content[:1000],
)
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=128,
messages=[{"role": "user", "content": prompt}],
)
raw = response.content[0].text.strip()
try:
import json
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
return (
bool(result.get("supports", False)),
float(result.get("confidence", 0.7)),
result.get("reason", ""),
)
except Exception:
return False, 0.5, "Parse error"
def check_all_citations(
self,
datapoint: RAGEvalDatapoint,
) -> CitationReport:
"""
Verify all citations in the datapoint.
Citations format: {claim_text: chunk_id}
"""
if not datapoint.citations:
return CitationReport(
total_citations=0,
accurate_citations=0,
citation_accuracy=1.0,
inaccurate=[],
)
chunk_lookup = {
chunk.chunk_id: chunk
for chunk in datapoint.retrieved_chunks
}
total = 0
accurate = 0
inaccurate = []
for claim, chunk_id in datapoint.citations.items():
total += 1
chunk = chunk_lookup.get(chunk_id)
if chunk is None:
# Citation references a chunk that was not retrieved - fabrication
inaccurate.append({
"claim": claim,
"cited_chunk_id": chunk_id,
"reason": "Chunk not in retrieved context - fabricated citation",
})
continue
supports, confidence, reason = self.verify_citation(claim, chunk)
if supports and confidence >= 0.6:
accurate += 1
else:
inaccurate.append({
"claim": claim,
"cited_chunk_id": chunk_id,
"reason": reason,
"confidence": confidence,
})
return CitationReport(
total_citations=total,
accurate_citations=accurate,
citation_accuracy=accurate / max(1, total),
inaccurate=inaccurate,
)
# ---------------------------------------------------------------------------
# Retrieval Evaluator
# ---------------------------------------------------------------------------
class RetrievalEvaluator:
"""
Computes standard information retrieval metrics for RAG retrieval quality.
Requires ground-truth relevance labels for retrieved chunks.
"""
def precision_at_k(
self,
retrieved: list[RetrievedChunk],
k: int,
) -> float:
"""Fraction of top-k retrieved chunks that are relevant."""
top_k = retrieved[:k]
if not top_k:
return 0.0
relevant_count = sum(
1 for chunk in top_k if chunk.is_relevant is True
)
return relevant_count / k
def recall_at_k(
self,
retrieved: list[RetrievedChunk],
all_relevant_ids: list[str],
k: int,
) -> float:
"""Fraction of all relevant chunks that appear in top-k."""
if not all_relevant_ids:
return 1.0 # No relevant chunks to miss
top_k_ids = {chunk.chunk_id for chunk in retrieved[:k]}
found = sum(1 for rid in all_relevant_ids if rid in top_k_ids)
return found / len(all_relevant_ids)
def ndcg_at_k(
self,
retrieved: list[RetrievedChunk],
k: int,
) -> float:
"""
Normalized Discounted Cumulative Gain at k.
Rewards retrieving relevant chunks at higher positions.
"""
top_k = retrieved[:k]
def dcg(chunks: list[RetrievedChunk]) -> float:
score = 0.0
for i, chunk in enumerate(chunks):
relevance = 1.0 if chunk.is_relevant else 0.0
score += relevance / math.log2(i + 2) # i+2 because i is 0-indexed
return score
actual_dcg = dcg(top_k)
# Ideal DCG: relevant chunks ranked first
ideal_chunks = sorted(
top_k,
key=lambda c: (1 if c.is_relevant else 0),
reverse=True,
)
ideal_dcg = dcg(ideal_chunks)
if ideal_dcg == 0:
return 1.0 # No relevant chunks - perfect by convention
return round(actual_dcg / ideal_dcg, 4)
def mrr(
self,
retrieved: list[RetrievedChunk],
) -> float:
"""
Mean Reciprocal Rank.
Returns 1/rank of the first relevant chunk, or 0 if none found.
"""
for i, chunk in enumerate(retrieved):
if chunk.is_relevant:
return 1.0 / (i + 1)
return 0.0
def hit_rate(
self,
retrieved: list[RetrievedChunk],
k: int,
) -> float:
"""Whether at least one relevant chunk appears in top-k (1 or 0)."""
top_k = retrieved[:k]
return 1.0 if any(c.is_relevant for c in top_k) else 0.0
def evaluate_retrieval(
self,
query: str,
retrieved_chunks: list[RetrievedChunk],
all_relevant_chunk_ids: list[str],
k: int = 5,
) -> RetrievalMetrics:
"""Compute all retrieval metrics for a single query."""
# Annotate chunks with relevance based on ground truth IDs
for chunk in retrieved_chunks:
if chunk.is_relevant is None:
chunk.is_relevant = chunk.chunk_id in all_relevant_chunk_ids
return RetrievalMetrics(
precision_at_k=self.precision_at_k(retrieved_chunks, k),
recall_at_k=self.recall_at_k(
retrieved_chunks, all_relevant_chunk_ids, k
),
ndcg_at_k=self.ndcg_at_k(retrieved_chunks, k),
mrr=self.mrr(retrieved_chunks),
hit_rate=self.hit_rate(retrieved_chunks, k),
k=k,
)
# ---------------------------------------------------------------------------
# Full RAG Evaluator
# ---------------------------------------------------------------------------
class FullRAGEvaluator:
"""
Comprehensive RAG evaluation combining all metrics.
Separates retrieval quality from generation quality.
"""
def __init__(self):
self.hallucination_classifier = HallucinationClassifier()
self.citation_evaluator = CitationAccuracyEvaluator()
self.retrieval_evaluator = RetrievalEvaluator()
def faithfulness(self, datapoint: RAGEvalDatapoint) -> float:
"""
Compute faithfulness: fraction of answer claims supported by context.
"""
verifications = self.hallucination_classifier.classify_all_claims(
datapoint.answer,
datapoint.question,
datapoint.retrieved_chunks,
)
if not verifications:
return 1.0
faithful_count = sum(1 for v in verifications if v.is_faithful)
return round(faithful_count / len(verifications), 4)
def answer_relevance(self, datapoint: RAGEvalDatapoint) -> float:
"""
Compute answer relevance via reverse question generation.
Generate N questions from the answer and measure similarity to original.
"""
n_reverse = 3
prompt = f"""Given this answer, generate {n_reverse} questions that this answer would be a good response to.
Answer: {datapoint.answer}
Output one question per line, no numbering:"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=256,
messages=[{"role": "user", "content": prompt}],
)
generated_questions = [
line.strip()
for line in response.content[0].text.strip().split("\n")
if line.strip()
]
if not generated_questions:
return 0.5
# Measure semantic similarity using embedding-like comparison via LLM judge
similarities = []
for gen_q in generated_questions[:n_reverse]:
sim_prompt = f"""On a scale of 0.0 to 1.0, how semantically similar are these two questions?
Question 1: {datapoint.question}
Question 2: {gen_q}
Output only a number between 0.0 and 1.0:"""
sim_response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=16,
messages=[{"role": "user", "content": sim_prompt}],
)
try:
sim = float(sim_response.content[0].text.strip())
sim = max(0.0, min(1.0, sim))
similarities.append(sim)
except ValueError:
similarities.append(0.5)
return round(statistics.mean(similarities), 4)
def context_precision(self, datapoint: RAGEvalDatapoint) -> float:
"""
Compute context precision: fraction of retrieved chunks that are relevant.
Uses LLM to judge relevance of each chunk to the question.
"""
if not datapoint.retrieved_chunks:
return 0.0
relevance_scores = []
for chunk in datapoint.retrieved_chunks:
prompt = f"""Is the following text chunk relevant to answering this question?
Question: {datapoint.question}
Chunk: {chunk.content[:500]}
Answer with JSON: {{"relevant": true/false, "confidence": 0.0-1.0}}"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
messages=[{"role": "user", "content": prompt}],
)
try:
import json
raw = response.content[0].text.strip()
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
relevance_scores.append(
1.0 if result.get("relevant", False) else 0.0
)
except Exception:
relevance_scores.append(0.5)
return round(statistics.mean(relevance_scores), 4)
def context_recall(self, datapoint: RAGEvalDatapoint) -> float:
"""
Compute context recall: what fraction of ground-truth statements
are covered by the retrieved context?
Requires datapoint.ground_truth to be set.
"""
if not datapoint.ground_truth:
return 1.0 # Cannot compute without reference
# Extract statements from ground truth
stmt_prompt = f"""Break this reference answer into a list of atomic factual statements.
Reference: {datapoint.ground_truth}
Output one statement per line, no numbering:"""
stmt_response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=512,
messages=[{"role": "user", "content": stmt_prompt}],
)
statements = [
line.strip()
for line in stmt_response.content[0].text.strip().split("\n")
if line.strip()
]
if not statements:
return 1.0
context_text = "\n\n".join(
f"[{c.chunk_id}] {c.content}" for c in datapoint.retrieved_chunks
)
# Check each statement against context
covered = 0
for stmt in statements:
check_prompt = f"""Is the following statement supported by the provided context?
Context: {context_text[:3000]}
Statement: {stmt}
Answer with JSON: {{"covered": true/false}}"""
check_response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=32,
messages=[{"role": "user", "content": check_prompt}],
)
try:
import json
raw = check_response.content[0].text.strip()
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
if result.get("covered", False):
covered += 1
except Exception:
pass
return round(covered / len(statements), 4)
def comprehensive_eval(
self,
datapoint: RAGEvalDatapoint,
all_relevant_chunk_ids: Optional[list[str]] = None,
k: int = 5,
) -> RAGEvalReport:
"""Run full evaluation suite on a single RAG datapoint."""
issues = []
# Core RAGAS metrics
faith = self.faithfulness(datapoint)
relevance = self.answer_relevance(datapoint)
precision = self.context_precision(datapoint)
recall = self.context_recall(datapoint)
# Hallucination breakdown
verifications = self.hallucination_classifier.classify_all_claims(
datapoint.answer,
datapoint.question,
datapoint.retrieved_chunks,
)
hallucination_breakdown = (
self.hallucination_classifier.compute_hallucination_rates(verifications)
)
# Citation accuracy
citation_report = None
if datapoint.citations:
citation_report = self.citation_evaluator.check_all_citations(datapoint)
if citation_report.citation_accuracy < 0.8:
issues.append(
f"Low citation accuracy: {citation_report.citation_accuracy:.0%}"
)
# Retrieval metrics
retrieval_metrics = None
if all_relevant_chunk_ids is not None:
retrieval_metrics = self.retrieval_evaluator.evaluate_retrieval(
datapoint.question,
datapoint.retrieved_chunks,
all_relevant_chunk_ids,
k=k,
)
if retrieval_metrics.precision_at_k < 0.5:
issues.append(
f"Low retrieval precision@{k}: "
f"{retrieval_metrics.precision_at_k:.0%}"
)
if retrieval_metrics.recall_at_k < 0.7:
issues.append(
f"Low retrieval recall@{k}: "
f"{retrieval_metrics.recall_at_k:.0%}"
)
# Issue detection
if faith < 0.7:
issues.append(
f"Low faithfulness ({faith:.0%}) - model may be using "
f"parametric memory instead of retrieved context"
)
if precision < 0.5:
issues.append(
f"Low context precision ({precision:.0%}) - too many "
f"irrelevant chunks being retrieved"
)
if hallucination_breakdown.get("inconsistency", 0) > 0.1:
issues.append(
"Inconsistency hallucinations detected - model contradicting "
"retrieved context"
)
# Overall score: weighted combination
overall = (
faith * 0.35
+ relevance * 0.25
+ precision * 0.20
+ recall * 0.20
)
return RAGEvalReport(
faithfulness=faith,
answer_relevance=relevance,
context_precision=precision,
context_recall=recall,
hallucination_breakdown=hallucination_breakdown,
citation_accuracy=(
citation_report.citation_accuracy if citation_report else None
),
retrieval_metrics=retrieval_metrics,
identified_issues=issues,
overall_score=round(overall, 4),
)
# ---------------------------------------------------------------------------
# Temporal Faithfulness Evaluator
# ---------------------------------------------------------------------------
class TemporalFaithfulnessEvaluator:
"""
Detects when a model uses an older retrieved document when a newer
one covering the same topic was also available.
Critical for documentation assistants where content changes over versions.
"""
def check_temporal_faithfulness(
self,
answer: str,
retrieved_chunks: list[RetrievedChunk],
) -> dict[str, bool | str]:
"""
Check whether the answer reflects the most recent retrieved information.
Returns {is_temporally_faithful, used_version, latest_available_version}.
"""
# Find chunks with version information
versioned = [c for c in retrieved_chunks if c.doc_version]
if len(versioned) < 2:
return {"is_temporally_faithful": True, "reason": "Single version"}
# Sort by version (simple string comparison - production would use semver)
versioned_sorted = sorted(versioned, key=lambda c: c.doc_version or "")
latest = versioned_sorted[-1]
# Ask LLM to identify which document version the answer draws from
chunks_with_versions = "\n\n".join(
f"[v{c.doc_version}] {c.content[:300]}"
for c in versioned_sorted
)
prompt = f"""An AI response was generated from multiple document versions.
Identify which version the answer primarily reflects.
Document versions available:
{chunks_with_versions}
AI Answer: {answer[:500]}
Which version does this answer primarily reflect?
JSON: {{"primary_version": "version string", "confidence": 0.0-1.0}}"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
messages=[{"role": "user", "content": prompt}],
)
try:
import json
raw = response.content[0].text.strip()
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
primary_version = result.get("primary_version", "unknown")
is_faithful = primary_version == latest.doc_version
return {
"is_temporally_faithful": is_faithful,
"used_version": primary_version,
"latest_available_version": latest.doc_version,
"reason": (
"Using latest version"
if is_faithful
else f"Used v{primary_version} when v{latest.doc_version} was available"
),
}
except Exception:
return {"is_temporally_faithful": True, "reason": "Could not determine"}
def detect_version_confusion(
self,
answer: str,
doc_versions: dict[str, str], # version_string -> content_summary
) -> Optional[dict]:
"""
Detect when the answer mixes information from multiple versions
in a way that creates internal contradictions.
"""
versions_text = "\n".join(
f"v{v}: {summary}"
for v, summary in doc_versions.items()
)
prompt = f"""Does this answer mix information from different API/product versions in a contradictory way?
Version information:
{versions_text}
Answer: {answer[:500]}
JSON: {{"has_version_confusion": true/false, "explanation": "brief reason"}}"""
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=128,
messages=[{"role": "user", "content": prompt}],
)
try:
import json
raw = response.content[0].text.strip()
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
if result.get("has_version_confusion", False):
return {
"detected": True,
"explanation": result.get("explanation", ""),
}
except Exception:
pass
return None
# ---------------------------------------------------------------------------
# RAG System Benchmark
# ---------------------------------------------------------------------------
class RAGSystemBenchmark:
"""
Runs full evaluation benchmarks across a test set.
Supports comparison between two RAG system configurations.
"""
def __init__(self):
self.evaluator = FullRAGEvaluator()
def run_full_benchmark(
self,
test_set: list[RAGEvalDatapoint],
relevant_chunk_ids: Optional[list[list[str]]] = None,
k: int = 5,
) -> dict[str, float]:
"""
Run complete benchmark across all test datapoints.
Returns aggregate metrics.
"""
faithfulness_scores = []
relevance_scores = []
precision_scores = []
recall_scores = []
all_issues = []
for i, datapoint in enumerate(test_set):
relevant_ids = (
relevant_chunk_ids[i]
if relevant_chunk_ids and i < len(relevant_chunk_ids)
else None
)
report = self.evaluator.comprehensive_eval(
datapoint, relevant_ids, k=k
)
faithfulness_scores.append(report.faithfulness)
relevance_scores.append(report.answer_relevance)
precision_scores.append(report.context_precision)
recall_scores.append(report.context_recall)
all_issues.extend(report.identified_issues)
return {
"faithfulness": statistics.mean(faithfulness_scores),
"answer_relevance": statistics.mean(relevance_scores),
"context_precision": statistics.mean(precision_scores),
"context_recall": statistics.mean(recall_scores),
"n_issues": len(all_issues),
"n_datapoints": len(test_set),
}
def compare_systems(
self,
system_a_results: dict[str, float],
system_b_results: dict[str, float],
) -> ComparisonReport:
"""Compare two RAG system benchmark results."""
improvements = {}
regressions = {}
for metric in system_a_results:
if metric in ("n_issues", "n_datapoints"):
continue
delta = system_b_results.get(metric, 0) - system_a_results.get(metric, 0)
if delta > 0.01:
improvements[metric] = round(delta, 4)
elif delta < -0.01:
regressions[metric] = round(delta, 4)
if not regressions:
recommendation = "APPROVE - System B improves on all metrics"
elif not improvements:
recommendation = "REJECT - System B regresses on all metrics"
elif regressions.get("faithfulness", 0) < -0.05:
recommendation = "REJECT - Critical faithfulness regression"
else:
recommendation = (
"REVIEW_NEEDED - Mixed results, human review required"
)
return ComparisonReport(
system_a_scores=system_a_results,
system_b_scores=system_b_results,
improvements=improvements,
regressions=regressions,
recommendation=recommendation,
)
def identify_weak_points(
self, results: dict[str, float]
) -> list[WeakPoint]:
"""Identify the metrics with the most room for improvement."""
weak_points = []
thresholds = {
"faithfulness": (0.8, "Increase prompt faithfulness constraints"),
"answer_relevance": (0.7, "Review response completeness"),
"context_precision": (0.6, "Tune retrieval to reduce irrelevant chunks"),
"context_recall": (0.7, "Increase retrieval top-k or improve chunking"),
}
for metric, (threshold, fix) in thresholds.items():
score = results.get(metric, 1.0)
if score < threshold:
weak_points.append(WeakPoint(
metric=metric,
score=score,
description=f"{metric} is below threshold {threshold} (actual: {score:.3f})",
recommended_fix=fix,
))
return sorted(weak_points, key=lambda w: w.score)
# ---------------------------------------------------------------------------
# Demo
# ---------------------------------------------------------------------------
if __name__ == "__main__":
# Simulate a RAG datapoint
datapoint = RAGEvalDatapoint(
question="What authentication methods does the API support?",
answer=(
"The API supports OAuth 2.0 and API key authentication. "
"Basic authentication was deprecated in v2.0 and should no longer be used."
),
retrieved_chunks=[
RetrievedChunk(
chunk_id="doc-001",
content=(
"The API supports two authentication methods: OAuth 2.0 for "
"user-delegated access and API key authentication for server-to-server "
"communication. Basic authentication was deprecated in v2.0."
),
source_doc="api-reference.md",
doc_version="2.1",
relevance_score=0.95,
),
RetrievedChunk(
chunk_id="doc-002",
content=(
"Rate limiting: API key requests are limited to 1000/hour. "
"OAuth requests follow the plan limits set at subscription time."
),
source_doc="rate-limits.md",
doc_version="2.1",
relevance_score=0.45,
),
],
ground_truth=(
"The API supports OAuth 2.0 and API key authentication. "
"Basic authentication was deprecated in v2.0."
),
)
evaluator = FullRAGEvaluator()
report = evaluator.comprehensive_eval(
datapoint,
all_relevant_chunk_ids=["doc-001"],
k=2,
)
print("=== RAG Evaluation Report ===")
print(f"Faithfulness: {report.faithfulness:.3f}")
print(f"Answer Relevance: {report.answer_relevance:.3f}")
print(f"Context Precision: {report.context_precision:.3f}")
print(f"Context Recall: {report.context_recall:.3f}")
print(f"Overall Score: {report.overall_score:.3f}")
if report.retrieval_metrics:
rm = report.retrieval_metrics
print(f"\nRetrieval Metrics (k={rm.k}):")
print(f" Precision@k: {rm.precision_at_k:.3f}")
print(f" Recall@k: {rm.recall_at_k:.3f}")
print(f" nDCG@k: {rm.ndcg_at_k:.3f}")
print(f" MRR: {rm.mrr:.3f}")
print(f" Hit Rate: {rm.hit_rate:.3f}")
if report.identified_issues:
print(f"\nIssues:")
for issue in report.identified_issues:
print(f" - {issue}")
Architecture Diagrams
RAG Evaluation Components
Hallucination Type Taxonomy
Citation Verification Pipeline
Production Notes
:::danger RAG Evaluation Without Decomposition Is Misleading Running only a final answer quality metric on a RAG system tells you almost nothing useful about what to fix. A high faithfulness score with low context precision means your retriever is polluting the context with junk. A high context precision with low faithfulness means your model is ignoring good context. You must measure each stage separately to know which stage is the bottleneck. :::
:::warning Context Window Pollution When context precision is below 0.5 (fewer than half of retrieved chunks are relevant), the irrelevant chunks are actively harming generation quality. They consume tokens, introduce off-topic information, and can anchor the model toward irrelevant content. Tune your retriever's reranking step - a cross-encoder reranker that scores each chunk for relevance to the specific query can dramatically improve context precision with no changes to the underlying retrieval index. :::
:::tip Faithfulness Threshold Selection For a documentation assistant, faithfulness below 0.9 is typically unacceptable - every out-of-context claim is potentially directing users to do something wrong. For a creative writing assistant where the retrieved context is used for inspiration rather than factual grounding, a faithfulness threshold of 0.5 may be appropriate. Set thresholds based on the downstream cost of hallucination in your specific domain, not on generic benchmarks. :::
:::note Evaluation Cost Management Running full RAGAS metrics with LLM-as-judge on 1,000 test examples costs roughly $5–15 depending on the judge model and average response length. For CI pipelines, use a two-tier approach: run the cheap retrieval metrics (precision@k, recall@k) on every change, and run the expensive generation metrics (faithfulness, context recall) on a random 10% sample or only on changes that touch the prompt or model version. :::
Interview Questions and Answers
Q1: What is faithfulness in RAG evaluation and how is it different from factual accuracy?
Faithfulness is a narrower, more measurable concept than factual accuracy. Factual accuracy asks "is this claim true according to world knowledge?" - a question that often requires a human domain expert to verify. Faithfulness asks "is this claim supported by the specific documents that were retrieved for this query?" - a question that can be automated by having an LLM compare the claim against the retrieved context. A claim can be factually accurate (correct according to the world) but unfaithful (not in the retrieved context, meaning the model used parametric knowledge instead of the retrieved source). In a medical or legal RAG system, unfaithfulness is critical even when the claim happens to be correct, because the citation chain is broken and users cannot verify the source.
Q2: How do you diagnose whether a RAG failure is a retrieval problem or a generation problem?
The key is to evaluate each stage independently. First, evaluate retrieval quality: given the query, does the retriever return documents that contain the information needed to answer it? Measure context recall (are the right documents retrieved?) and context precision (are most retrieved documents relevant?). If context recall is low, the retriever is the bottleneck. Next, given that the right documents were retrieved, evaluate generation: does the answer faithfully reflect what was in those documents? Measure faithfulness (are answer claims supported by context?). If faithfulness is low despite good retrieval, the model is the bottleneck - it may be over-relying on parametric memory. This two-stage diagnosis tells you exactly which component to fix and prevents wasted effort on the wrong subsystem.
Q3: What are the three types of hallucinations in RAG systems and what causes each?
Out-of-context hallucinations occur when the model uses knowledge from pre-training that is not in the retrieved documents. The model has strong priors from training and sometimes generates from those priors instead of the provided context - especially for topics it has seen frequently in training data. Fix: stronger prompt instructions to use only the context, or a model fine-tuned for grounded generation. Inconsistency hallucinations occur when the model generates a claim that directly contradicts something in the retrieved context. This is the most dangerous type - the model is actively fighting against the provided information. Causes include conflicting documents in the context, ambiguous phrasing, or model alignment failures. Fix: deduplication and conflict detection in the retrieved context, plus model quality improvements. Fabrication hallucinations occur when the model invents citations, document names, or sources that don't exist. Fix: strict output format enforcement and post-processing citation validation.
Q4: How does nDCG differ from Precision@k for retrieval evaluation, and when does the difference matter?
Precision@k measures the fraction of retrieved chunks that are relevant, treating all positions equally. nDCG additionally rewards systems that put relevant chunks first. The difference matters when your generation model uses positional attention differently across context positions - many LLMs attend more strongly to content at the beginning and end of the context window (the "lost in the middle" phenomenon). If your model reads the first retrieved chunk most carefully, a system that ranks the relevant chunk first (high nDCG) will perform better in practice than a system with the same Precision@k that buries the relevant chunk in position 4 of 5. Use nDCG as your primary retrieval metric when you know your model is sensitive to chunk ordering.
Q5: How do you handle temporal faithfulness - ensuring the model uses the most recent documentation when multiple versions are retrieved?
Temporal faithfulness failures occur in documentation assistants where the corpus contains documents from multiple product versions and the retriever returns chunks from different versions for the same query. The fix is multi-layered: at retrieval time, add a recency bias to the ranker so that newer documents score higher for queries that don't explicitly ask about a specific version. At indexing time, tag every chunk with a version label and remove or archive deprecated content rather than keeping it in the live index. At evaluation time, when you have datapoints where the answer should reflect a specific version, add a temporal faithfulness check that identifies which version the generated answer draws from and flags cases where an older version was used when a newer one was available.
Q6: What is context recall and why does computing it require a reference answer?
Context recall measures what fraction of the information needed to answer a question is present in the retrieved context. To compute it, you need to know what information is required - which means you need a reference answer that contains all the necessary facts. You extract atomic statements from the reference answer and check each one against the retrieved context. If 8 out of 10 required statements are covered by the retrieved chunks, context recall is 0.8. Without a reference answer, you cannot know what information was needed in the first place. This is why context recall is more expensive to compute than context precision - it requires human-annotated golden answers or high-quality LLM-generated references, not just a relevance judgment on the retrieved chunks.