:::tip 🎮 Interactive Playground Visualize this concept: Try the Prompt Version Management demo on the EngineersOfAI Playground - no code required. :::
Regression Testing for Prompts
Reading time: 38 minutes | Interview relevance: Very High | Target roles: AI Engineer, ML Engineer, Platform Engineer, MLOps
The Silent Regression
The engineering team had been improving their system prompt for three weeks. The task was mundane: their AI assistant was sometimes ambiguous when users asked questions in Spanish mixed with English - code-switching queries that were common in their user base. The fix seemed obvious after a careful read: one sentence in the system prompt was implicitly assuming English-only input. Clarifying that sentence to explicitly welcome multi-language queries took two minutes.
The spot-check looked good. They tested the change on 20 diverse questions including five in Spanish, three in French, and two that mixed languages. All twenty passed. The response quality was clearly better for multilingual input. They merged and deployed on a Thursday afternoon.
The following Monday, a customer success manager forwarded a thread of support tickets. A pattern had emerged over the weekend: the assistant was now producing responses with inconsistent capitalization rules for proper nouns, and in a specific edge case - questions about compound technical terms that contained acronyms - the responses were using inconsistent casing in a way that made them look unprofessional. This edge case accounted for roughly 8% of their production traffic.
The regression had nothing to do with multilingual input. The clarified sentence had subtly shifted how the model interpreted formatting instructions elsewhere in the prompt. By removing an implicit English-language assumption, the system had also slightly de-prioritized English typography conventions. The interaction was invisible on a 20-case spot check. It was only visible across 8% of production traffic over three days.
The team spent two days diagnosing the issue, one day writing and deploying the fix, and an entire post-mortem meeting explaining to leadership why they hadn't caught it before deployment. The total cost was approximately 40 engineer-hours and measurable user frustration. The lesson was simple but expensive: spot checks test forward progress, not backward coverage. Without a regression test suite that covered the full range of previously working behaviors, every prompt change was flying blind.
Why Regression Testing for Prompts
Prompts are code. Every change to a system prompt is a change to the program that controls your AI system's behavior. Software engineers understood decades ago that code changes can break previously working behavior, which is why regression testing is standard practice in software development. The same principle applies to prompts - with one complication: prompt behavior is probabilistic, which means regression testing must account for non-determinism in a way that traditional software testing does not.
Silent regressions are the expensive ones. The most costly bugs are not the ones that crash the system - those are immediately visible. The costly ones are the ones that degrade quality in a subset of traffic that no one is actively monitoring. An 8% regression in a specific query type might never surface in a 20-case spot check. It surfaces in production three days later, after thousands of users have received degraded responses.
The fix-one-break-two problem. As system prompts grow more complex - adding handling for edge cases, formatting rules, safety constraints, persona instructions - the interaction effects between different parts of the prompt multiply. Changing one section to address a known failure can trigger a failure in a previously working section. Without a regression suite, you have no systematic way to detect these interactions.
Prompt complexity compounds. A simple prompt has few failure modes. A production prompt that has been refined over months - with sections for tone, format, safety, domain constraints, language handling, output structure - has hundreds of failure modes from interactions between sections. The regression suite must grow alongside the prompt complexity.
Historical Context
Regression testing as a formal practice comes from software engineering circa the 1970s, when the term was coined to describe testing that verifies previously fixed bugs stay fixed. The discipline evolved through the 1980s and 1990s as test automation tools matured - unit testing frameworks, continuous integration systems, and test coverage measurement became standard.
The application of regression testing to machine learning systems is newer. ML regression testing emerged prominently in the 2016–2019 period as companies building production ML products discovered that model updates could silently degrade performance on specific subpopulations. The solution - maintaining "behavioral test suites" that checked model outputs on a curated set of cases - was described systematically by Ribeiro et al. in the CheckList paper (2020), which introduced the concept of behavioral testing for NLP models.
The extension to prompts specifically is a post-2022 development, emerging as teams building LLM-powered products discovered that prompt changes had the same regression risk as code changes. By 2023, companies shipping LLM products at scale had developed internal prompt regression frameworks, and by 2024 the practice had become standard enough that multiple open-source frameworks (PromptFoo, EvalAI, Braintrust) included regression testing as a first-class feature.
What to Test
Happy Path Cases
The core 20–30 use cases the system absolutely must handle correctly, representing the primary user workflows. For a coding assistant: "write a function that does X", "explain this code", "fix this bug". These should always pass - if they fail after a prompt change, the change is categorically unshippable.
Edge Cases
Inputs at the extremes of valid input space: empty input, very long input (near context limit), unusual formatting (all caps, no punctuation, heavily formatted markdown), single-character input, input that is valid but unusual.
Adversarial Cases
Inputs that have previously broken the system: prompt injection attempts, questions designed to elicit refusals, inputs that trigger formatting failures. Every production failure that required a prompt fix becomes an adversarial regression test.
Domain Boundary Cases
Inputs at the edge of what the system should handle - just inside and just outside the intended domain. For a customer support assistant: technical questions that are just slightly too complex for first-line support (should escalate), questions about competitors (should handle gracefully), questions about topics completely outside the domain (should decline or redirect).
Regression Cases
Every documented production failure gets a corresponding regression test. This is the most valuable category: these are known failure modes with confirmed fixes. If the fix breaks, you want to know immediately - not three days later via a customer escalation.
Test Case Design
Specification Format
A regression test specifies:
- Input: the exact user message (and any relevant context variables)
- Expected properties: what the response MUST contain or exhibit
- Forbidden properties: what the response MUST NOT contain or exhibit
- Pass/fail criteria: how to determine if the test passed
- Severity: CRITICAL (blocks deployment), HIGH, MEDIUM, LOW
- Source: where this test came from (production failure, edge case, adversarial, happy path)
Note the design principle: expected properties are behavioral specifications, not exact string matches. An exact string match would make regression tests brittle - a response that is semantically equivalent but worded differently would fail. Instead, specify properties like "response acknowledges the user's question", "response does not claim to be human", "response includes a code example".
Determinism Strategy
LLM outputs are probabilistic. The same prompt will produce slightly different outputs across runs. Regression testing must handle this:
- Lower temperature: use temperature=0 or temperature=0.1 for test runs to reduce variance
- Multiple runs: run each test 3–5 times and use the majority result
- Tolerance threshold: mark a test as PASS if it passes 70%+ of runs, FLAKY if 30–70%, FAIL if below 30%
Test Coverage Metrics
Coverage for prompt regression testing is not about code line coverage - it's about production traffic coverage. What fraction of the query types your users actually submit are represented in your test suite? Measure this by computing the distribution of production queries (by topic, type, length) and comparing it to your test suite's distribution. Gaps in coverage are gaps in regression protection.
Complete Implementation
import anthropic
import asyncio
import hashlib
import json
import statistics
from dataclasses import dataclass, field
from typing import Optional, Callable
from enum import Enum
from collections import defaultdict, Counter
from datetime import datetime, timezone
client = anthropic.Anthropic()
async_client = anthropic.AsyncAnthropic()
# ---------------------------------------------------------------------------
# Data Structures
# ---------------------------------------------------------------------------
class TestResult(Enum):
PASS = "PASS"
FAIL = "FAIL"
FLAKY = "FLAKY"
ERROR = "ERROR"
class Severity(Enum):
CRITICAL = "CRITICAL"
HIGH = "HIGH"
MEDIUM = "MEDIUM"
LOW = "LOW"
class TestSource(Enum):
PRODUCTION_FAILURE = "production_failure"
EDGE_CASE = "edge_case"
ADVERSARIAL = "adversarial"
HAPPY_PATH = "happy_path"
DOMAIN_BOUNDARY = "domain_boundary"
@dataclass
class RegressionTest:
"""A single regression test case."""
test_id: str
description: str
user_message: str
input_variables: dict = field(default_factory=dict)
expected_properties: list[str] = field(default_factory=list)
forbidden_properties: list[str] = field(default_factory=list)
source: TestSource = TestSource.HAPPY_PATH
severity: Severity = Severity.MEDIUM
created_at: str = field(
default_factory=lambda: datetime.now(timezone.utc).isoformat()
)
metadata: dict = field(default_factory=dict)
def render_message(self) -> str:
"""Render user_message with input_variables substituted."""
msg = self.user_message
for key, value in self.input_variables.items():
msg = msg.replace(f"{{{key}}}", str(value))
return msg
@dataclass
class SingleRunResult:
"""Result of a single test execution."""
passed: bool
response_text: str
failed_expected: list[str] # Expected properties that were missing
failed_forbidden: list[str] # Forbidden properties that were present
error: Optional[str] = None
@dataclass
class TestRunResult:
"""Aggregated result across multiple runs."""
test_id: str
result: TestResult
pass_rate: float
runs: list[SingleRunResult]
response_sample: str # Representative response for debugging
@dataclass
class SuiteResult:
"""Result of running a full regression suite."""
total: int
passed: int
failed: int
flaky: int
errors: int
critical_failures: list[str] # test_ids
pass_rate: float
run_duration_seconds: float
cost_estimate_usd: float
results: list[TestRunResult]
@dataclass
class DiffReport:
"""Comparison between two prompt versions on the same test suite."""
improved_tests: list[str] # test_ids that went FAIL -> PASS
regressed_tests: list[str] # test_ids that went PASS -> FAIL
unchanged_tests: list[str] # test_ids with same result
critical_regressions: list[str] # regressed CRITICAL severity tests
recommendation: str # APPROVE | REJECT | REVIEW_NEEDED
@dataclass
class CoverageReport:
"""How well the test suite covers production traffic patterns."""
production_topic_dist: dict[str, float]
suite_topic_dist: dict[str, float]
coverage_gaps: list[str] # Topics in production but not in suite
coverage_score: float # 0-1, 1 = perfect alignment
recommendations: list[str]
@dataclass
class FlakinessReport:
"""Analysis of a single test's non-determinism."""
test_id: str
pass_rate: float
classification: str # "broken" | "flaky" | "passing"
recommended_fix: str
# ---------------------------------------------------------------------------
# LLM Judge Evaluator
# ---------------------------------------------------------------------------
class LLMJudgeEvaluator:
"""
Uses claude-haiku to evaluate whether a response satisfies
expected and forbidden properties.
"""
PROPERTY_CHECK_PROMPT = """Evaluate whether the following AI response satisfies all the given properties.
User message: {user_message}
AI Response:
{response}
Properties to check:
{properties}
For each property, determine if it is satisfied.
Respond with JSON: {{"results": [{{"property": "...", "satisfied": true/false, "reason": "brief"}}]}}"""
def check_properties(
self,
user_message: str,
response: str,
properties: list[str],
expect_satisfied: bool,
) -> list[str]:
"""
Check list of properties against response.
Returns list of properties that FAILED the check:
- If expect_satisfied=True: returns properties that were NOT satisfied
- If expect_satisfied=False: returns properties that WERE satisfied (forbidden)
"""
if not properties:
return []
properties_text = "\n".join(f"- {p}" for p in properties)
prompt = self.PROPERTY_CHECK_PROMPT.format(
user_message=user_message,
response=response[:2000],
properties=properties_text,
)
api_response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=512,
messages=[{"role": "user", "content": prompt}],
)
raw = api_response.content[0].text.strip()
try:
start = raw.index("{")
end = raw.rindex("}") + 1
result = json.loads(raw[start:end])
results = result.get("results", [])
except Exception:
# If parsing fails, assume all passed to avoid false failures
return []
failed = []
for item in results:
prop = item.get("property", "")
satisfied = item.get("satisfied", True)
if expect_satisfied and not satisfied:
failed.append(prop)
elif not expect_satisfied and satisfied:
failed.append(prop)
return failed
def evaluate(
self,
test: RegressionTest,
response: str,
) -> SingleRunResult:
"""Evaluate one response against a test's pass/fail criteria."""
# Check expected properties (must be present)
failed_expected = self.check_properties(
test.render_message(),
response,
test.expected_properties,
expect_satisfied=True,
)
# Check forbidden properties (must not be present)
failed_forbidden = self.check_properties(
test.render_message(),
response,
test.forbidden_properties,
expect_satisfied=False,
)
passed = not failed_expected and not failed_forbidden
return SingleRunResult(
passed=passed,
response_text=response,
failed_expected=failed_expected,
failed_forbidden=failed_forbidden,
)
# ---------------------------------------------------------------------------
# Prompt Test Runner
# ---------------------------------------------------------------------------
class PromptTestRunner:
"""
Executes regression tests against a system prompt, with support for
multiple runs (for flakiness detection), parallel execution, and
budget-constrained running.
"""
# Approximate cost per 1K tokens (claude-haiku for judge, opus for subject)
COST_PER_1K_INPUT = 0.000003 # claude-haiku
COST_PER_1K_OUTPUT = 0.000015 # claude-haiku
def __init__(self):
self.judge = LLMJudgeEvaluator()
def _run_once(
self,
test: RegressionTest,
system_prompt: str,
model: str = "claude-opus-4-6",
temperature: float = 0.1,
) -> SingleRunResult:
"""Execute the test once and evaluate the response."""
try:
response = client.messages.create(
model=model,
max_tokens=1024,
temperature=temperature,
system=system_prompt,
messages=[{"role": "user", "content": test.render_message()}],
)
response_text = response.content[0].text
except Exception as e:
return SingleRunResult(
passed=False,
response_text="",
failed_expected=[],
failed_forbidden=[],
error=str(e),
)
return self.judge.evaluate(test, response_text)
def run_test(
self,
test: RegressionTest,
system_prompt: str,
n_runs: int = 3,
model: str = "claude-opus-4-6",
temperature: float = 0.1,
) -> TestRunResult:
"""Run a test n_runs times and aggregate results."""
runs = [
self._run_once(test, system_prompt, model, temperature)
for _ in range(n_runs)
]
pass_count = sum(1 for r in runs if r.passed)
pass_rate = pass_count / n_runs
if pass_rate >= 0.7:
result = TestResult.PASS
elif pass_rate < 0.3:
result = TestResult.FAIL
else:
result = TestResult.FLAKY
# Find a representative failing response for debugging
failing_runs = [r for r in runs if not r.passed]
sample_response = (
failing_runs[0].response_text if failing_runs else runs[0].response_text
)
return TestRunResult(
test_id=test.test_id,
result=result,
pass_rate=pass_rate,
runs=runs,
response_sample=sample_response[:500],
)
async def _run_once_async(
self,
test: RegressionTest,
system_prompt: str,
model: str,
temperature: float,
) -> SingleRunResult:
"""Async version for parallel execution."""
try:
response = await async_client.messages.create(
model=model,
max_tokens=1024,
temperature=temperature,
system=system_prompt,
messages=[{"role": "user", "content": test.render_message()}],
)
response_text = response.content[0].text
except Exception as e:
return SingleRunResult(
passed=False,
response_text="",
failed_expected=[],
failed_forbidden=[],
error=str(e),
)
return self.judge.evaluate(test, response_text)
async def run_suite_async(
self,
suite: list[RegressionTest],
system_prompt: str,
n_runs: int = 3,
model: str = "claude-opus-4-6",
temperature: float = 0.1,
) -> list[TestRunResult]:
"""Run all tests in a suite concurrently."""
import time
start = time.time()
async def run_one(test: RegressionTest) -> TestRunResult:
runs = await asyncio.gather(*[
self._run_once_async(test, system_prompt, model, temperature)
for _ in range(n_runs)
])
pass_count = sum(1 for r in runs if r.passed)
pass_rate = pass_count / n_runs
result = (
TestResult.PASS if pass_rate >= 0.7
else TestResult.FAIL if pass_rate < 0.3
else TestResult.FLAKY
)
failing = [r for r in runs if not r.passed]
sample = failing[0].response_text if failing else runs[0].response_text
return TestRunResult(
test_id=test.test_id,
result=result,
pass_rate=pass_rate,
runs=list(runs),
response_sample=sample[:500],
)
all_results = await asyncio.gather(*[run_one(t) for t in suite])
return list(all_results)
def run_suite(
self,
suite: list[RegressionTest],
system_prompt: str,
n_runs: int = 3,
parallel: bool = True,
model: str = "claude-opus-4-6",
) -> SuiteResult:
"""Run a full suite, optionally in parallel."""
import time
start = time.time()
if parallel:
results = asyncio.run(
self.run_suite_async(suite, system_prompt, n_runs, model)
)
else:
results = [
self.run_test(t, system_prompt, n_runs, model)
for t in suite
]
duration = time.time() - start
cost = self.estimate_cost(suite, model)
# Build severity lookup
severity_lookup = {t.test_id: t.severity for t in suite}
passed = sum(1 for r in results if r.result == TestResult.PASS)
failed = sum(1 for r in results if r.result == TestResult.FAIL)
flaky = sum(1 for r in results if r.result == TestResult.FLAKY)
errors = sum(
1 for r in results if any(run.error for run in r.runs)
)
critical_failures = [
r.test_id
for r in results
if r.result == TestResult.FAIL
and severity_lookup.get(r.test_id) == Severity.CRITICAL
]
return SuiteResult(
total=len(suite),
passed=passed,
failed=failed,
flaky=flaky,
errors=errors,
critical_failures=critical_failures,
pass_rate=passed / max(1, len(suite)),
run_duration_seconds=round(duration, 2),
cost_estimate_usd=cost["usd"],
results=results,
)
def estimate_cost(
self,
suite: list[RegressionTest],
model: str = "claude-opus-4-6",
) -> dict[str, float]:
"""Estimate API cost before running the suite."""
# Rough token estimates
avg_input_tokens = 500 # system prompt + test message
avg_output_tokens = 300 # typical response
n_tests = len(suite)
n_runs = 3
n_judge_calls = 2 # one for expected, one for forbidden
subject_calls = n_tests * n_runs
judge_calls = n_tests * n_runs * n_judge_calls
total_input_tokens = (
subject_calls * avg_input_tokens
+ judge_calls * 400
)
total_output_tokens = (
subject_calls * avg_output_tokens
+ judge_calls * 100
)
usd = (
total_input_tokens / 1000 * self.COST_PER_1K_INPUT
+ total_output_tokens / 1000 * self.COST_PER_1K_OUTPUT
)
return {
"usd": round(usd, 4),
"input_tokens": total_input_tokens,
"output_tokens": total_output_tokens,
"subject_api_calls": subject_calls,
"judge_api_calls": judge_calls,
}
def run_with_budget(
self,
suite: list[RegressionTest],
system_prompt: str,
max_usd: float = 5.0,
model: str = "claude-opus-4-6",
) -> SuiteResult:
"""
Run as many tests as budget allows, prioritizing by severity.
Critical tests always run first.
"""
# Sort by severity (CRITICAL first)
severity_order = {
Severity.CRITICAL: 0,
Severity.HIGH: 1,
Severity.MEDIUM: 2,
Severity.LOW: 3,
}
sorted_suite = sorted(suite, key=lambda t: severity_order[t.severity])
# Estimate per-test cost
per_test_usd = self.estimate_cost([sorted_suite[0]], model)["usd"]
max_tests = min(len(sorted_suite), int(max_usd / max(per_test_usd, 0.001)))
print(f"Budget ${max_usd:.2f} allows {max_tests}/{len(sorted_suite)} tests")
return self.run_suite(
sorted_suite[:max_tests], system_prompt, parallel=True, model=model
)
# ---------------------------------------------------------------------------
# Regression Cache
# ---------------------------------------------------------------------------
class RegressionCache:
"""
Caches test results keyed by (test_id, prompt_hash).
Avoids re-running tests when the prompt hasn't changed.
"""
def __init__(self, cache_path: str = "/tmp/regression_cache.json"):
self.cache_path = cache_path
self._cache: dict[str, dict] = self._load()
def _load(self) -> dict:
try:
with open(self.cache_path) as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return {}
def _save(self) -> None:
with open(self.cache_path, "w") as f:
json.dump(self._cache, f, indent=2)
def prompt_hash(self, system_prompt: str) -> str:
return hashlib.md5(system_prompt.encode()).hexdigest()[:16]
def _cache_key(self, test_id: str, prompt_hash: str) -> str:
return f"{test_id}::{prompt_hash}"
def get_cached_result(
self, test_id: str, prompt_hash: str
) -> Optional[TestRunResult]:
key = self._cache_key(test_id, prompt_hash)
cached = self._cache.get(key)
if cached is None:
return None
# Reconstruct TestRunResult from cached dict
return TestRunResult(
test_id=cached["test_id"],
result=TestResult(cached["result"]),
pass_rate=cached["pass_rate"],
runs=[], # Don't store individual runs
response_sample=cached["response_sample"],
)
def cache_result(
self, test_id: str, prompt_hash: str, result: TestRunResult
) -> None:
key = self._cache_key(test_id, prompt_hash)
self._cache[key] = {
"test_id": result.test_id,
"result": result.result.value,
"pass_rate": result.pass_rate,
"response_sample": result.response_sample,
"cached_at": datetime.now(timezone.utc).isoformat(),
}
self._save()
def invalidate_on_prompt_change(
self, old_hash: str, new_hash: str
) -> int:
"""Remove cached results from old prompt hash. Returns count invalidated."""
keys_to_remove = [
k for k in self._cache if k.endswith(f"::{old_hash}")
]
for k in keys_to_remove:
del self._cache[k]
if keys_to_remove:
self._save()
return len(keys_to_remove)
# ---------------------------------------------------------------------------
# Flaky Test Detector
# ---------------------------------------------------------------------------
class FlakyTestDetector:
"""
Identifies tests that produce inconsistent results due to LLM non-determinism.
Flaky tests are a quality signal - they indicate under-specified behavior.
"""
def __init__(self):
self.runner = PromptTestRunner()
def run_multiple(
self,
test: RegressionTest,
system_prompt: str,
n: int = 10,
) -> FlakinessReport:
"""Run a test n times to measure its flakiness."""
run_results = [
self.runner._run_once(test, system_prompt)
for _ in range(n)
]
pass_count = sum(1 for r in run_results if r.passed)
pass_rate = pass_count / n
if pass_rate < 0.3:
classification = "broken"
elif pass_rate <= 0.7:
classification = "flaky"
else:
classification = "passing"
fix = self._recommend_fix(test, pass_rate)
return FlakinessReport(
test_id=test.test_id,
pass_rate=pass_rate,
classification=classification,
recommended_fix=fix,
)
def _recommend_fix(
self, test: RegressionTest, pass_rate: float
) -> str:
if pass_rate < 0.3:
return (
"Test is consistently failing - check if the system prompt "
"actually supports this behavior"
)
msg_length = len(test.user_message)
if msg_length > 200:
return (
"Long input increases variance - add explicit output format "
"constraints or split into smaller tests"
)
if len(test.expected_properties) > 5:
return (
"Too many expected properties in one test - split into "
"separate focused tests"
)
if any(
vague in " ".join(test.expected_properties).lower()
for vague in ["good", "appropriate", "helpful", "nice"]
):
return (
"Expected properties are too vague - rewrite with specific, "
"measurable criteria"
)
return (
"Lower the model temperature to 0.0 for this test, or add "
"explicit constraints in the system prompt"
)
# ---------------------------------------------------------------------------
# Regression Suite Manager
# ---------------------------------------------------------------------------
class RegressionSuiteManager:
"""
Builds and maintains the regression test suite over time,
converting production failures into test cases.
"""
def add_from_production_failure(
self,
failure_case: dict,
) -> RegressionTest:
"""
Convert a documented production failure into a regression test.
failure_case: {user_message, failing_behavior, expected_behavior,
severity, topic}
"""
test_id = "reg_" + hashlib.md5(
failure_case["user_message"].encode()
).hexdigest()[:8]
# The expected property is that the failing behavior does not occur
expected_properties = [failure_case["expected_behavior"]]
forbidden_properties = [failure_case["failing_behavior"]]
return RegressionTest(
test_id=test_id,
description=(
f"Regression: {failure_case.get('topic', 'production failure')}"
),
user_message=failure_case["user_message"],
expected_properties=expected_properties,
forbidden_properties=forbidden_properties,
source=TestSource.PRODUCTION_FAILURE,
severity=Severity[failure_case.get("severity", "HIGH")],
metadata={"original_failure": failure_case},
)
def import_from_eval_dataset(
self,
golden_examples: list[dict],
max_score_threshold: float = 3.0,
) -> list[RegressionTest]:
"""
Import regression tests from a golden evaluation dataset.
Only include examples where the system previously performed well.
Examples with scores <= threshold are treated as known failures, not regressions.
"""
tests = []
for ex in golden_examples:
current_score = ex.get("current_system_score", 5.0)
if current_score <= max_score_threshold:
continue # Already failing - not a regression baseline
test_id = "ds_" + ex.get("example_id", hashlib.md5(
ex["question"].encode()
).hexdigest()[:8])
tests.append(RegressionTest(
test_id=test_id,
description=f"Golden dataset: {ex.get('topic', 'general')}",
user_message=ex["question"],
expected_properties=[
"Response addresses the question asked",
"Response is factually consistent with the topic",
],
forbidden_properties=[
"Response refuses to answer without reason",
"Response is completely off-topic",
],
source=TestSource.HAPPY_PATH,
severity=Severity.MEDIUM,
))
return tests
def prioritize_suite(
self,
suite: list[RegressionTest],
deployment_type: str = "minor",
) -> list[RegressionTest]:
"""
Order tests by priority for the given deployment type.
'patch': critical only
'minor': critical + high
'major': all tests
"""
if deployment_type == "patch":
allowed_severities = {Severity.CRITICAL}
elif deployment_type == "minor":
allowed_severities = {Severity.CRITICAL, Severity.HIGH}
else:
allowed_severities = {
Severity.CRITICAL, Severity.HIGH, Severity.MEDIUM, Severity.LOW
}
severity_order = {
Severity.CRITICAL: 0,
Severity.HIGH: 1,
Severity.MEDIUM: 2,
Severity.LOW: 3,
}
filtered = [t for t in suite if t.severity in allowed_severities]
return sorted(filtered, key=lambda t: severity_order[t.severity])
def generate_coverage_report(
self,
suite: list[RegressionTest],
production_logs: list[dict],
) -> CoverageReport:
"""
Measure how well the test suite covers production query patterns.
"""
# Production topic distribution
prod_topics = Counter(
log.get("topic", "unknown") for log in production_logs
)
total_prod = sum(prod_topics.values())
prod_dist = {
t: c / total_prod for t, c in prod_topics.items()
}
# Suite topic distribution
suite_topics = Counter(
t.metadata.get("topic", "unknown") for t in suite
)
total_suite = max(1, sum(suite_topics.values()))
suite_dist = {
t: c / total_suite for t, c in suite_topics.items()
}
# Coverage gaps: production topics not covered in suite
gaps = [
topic
for topic, frac in prod_dist.items()
if frac > 0.05 and topic not in suite_dist
]
# Coverage score: 1 - average absolute difference in distribution
all_topics = set(prod_dist.keys()) | set(suite_dist.keys())
avg_diff = statistics.mean(
abs(prod_dist.get(t, 0) - suite_dist.get(t, 0))
for t in all_topics
)
coverage_score = max(0.0, 1.0 - avg_diff * 2)
recommendations = []
for gap in gaps:
frac = prod_dist.get(gap, 0)
recommendations.append(
f"Add {int(frac * total_suite * 0.1) + 1} tests for topic '{gap}' "
f"({frac:.0%} of production traffic)"
)
return CoverageReport(
production_topic_dist=prod_dist,
suite_topic_dist=suite_dist,
coverage_gaps=gaps,
coverage_score=round(coverage_score, 3),
recommendations=recommendations,
)
# ---------------------------------------------------------------------------
# Differential Tester
# ---------------------------------------------------------------------------
class DifferentialTester:
"""
Compares two prompt versions on the same test suite,
identifying improvements and regressions precisely.
"""
def __init__(self):
self.runner = PromptTestRunner()
def compare(
self,
prompt_v1: str,
prompt_v2: str,
suite: list[RegressionTest],
n_runs: int = 3,
) -> DiffReport:
"""
Run suite against both prompts and compare results.
"""
print(f"Running v1 baseline ({len(suite)} tests)...")
v1_suite = self.runner.run_suite(
suite, prompt_v1, n_runs=n_runs, parallel=True
)
print(f"Running v2 candidate ({len(suite)} tests)...")
v2_suite = self.runner.run_suite(
suite, prompt_v2, n_runs=n_runs, parallel=True
)
v1_map = {r.test_id: r for r in v1_suite.results}
v2_map = {r.test_id: r for r in v2_suite.results}
# Severity lookup
severity_lookup = {t.test_id: t.severity for t in suite}
improved = []
regressed = []
unchanged = []
critical_regressions = []
for test in suite:
v1_result = v1_map.get(test.test_id)
v2_result = v2_map.get(test.test_id)
if not v1_result or not v2_result:
continue
was_passing = v1_result.result == TestResult.PASS
now_passing = v2_result.result == TestResult.PASS
if not was_passing and now_passing:
improved.append(test.test_id)
elif was_passing and not now_passing:
regressed.append(test.test_id)
if severity_lookup.get(test.test_id) == Severity.CRITICAL:
critical_regressions.append(test.test_id)
else:
unchanged.append(test.test_id)
recommendation = self._recommend(
len(improved), len(regressed), critical_regressions
)
return DiffReport(
improved_tests=improved,
regressed_tests=regressed,
unchanged_tests=unchanged,
critical_regressions=critical_regressions,
recommendation=recommendation,
)
def _recommend(
self,
n_improved: int,
n_regressed: int,
critical_regressions: list[str],
) -> str:
if critical_regressions:
return (
f"REJECT - {len(critical_regressions)} CRITICAL regression(s): "
+ ", ".join(critical_regressions)
)
if n_regressed == 0:
return "APPROVE - no regressions detected"
if n_improved > n_regressed * 2:
return (
f"REVIEW_NEEDED - {n_improved} improvements vs {n_regressed} "
f"non-critical regressions - consider if trade-off is acceptable"
)
return (
f"REJECT - {n_regressed} regressions with only {n_improved} improvements"
)
# ---------------------------------------------------------------------------
# Pre-Deployment Gate
# ---------------------------------------------------------------------------
def pre_deploy_check(
old_prompt: str,
new_prompt: str,
suite: list[RegressionTest],
regression_threshold: float = 0.95,
) -> tuple[bool, str, DiffReport]:
"""
Full pre-deployment gate:
1. Run differential test (old vs new prompt)
2. Block on any CRITICAL regression
3. Block if overall pass rate drops below threshold
4. Return (should_deploy, reason, diff_report)
"""
tester = DifferentialTester()
runner = PromptTestRunner()
diff = tester.compare(old_prompt, new_prompt, suite)
# Hard block: any critical regression
if diff.critical_regressions:
return (
False,
f"Blocked: CRITICAL regressions in {diff.critical_regressions}",
diff,
)
# Soft block: overall pass rate dropped
critical_high_tests = [
t for t in suite
if t.severity in (Severity.CRITICAL, Severity.HIGH)
]
new_suite_result = runner.run_suite(
critical_high_tests, new_prompt, n_runs=3
)
if new_suite_result.pass_rate < regression_threshold:
return (
False,
f"Blocked: Pass rate {new_suite_result.pass_rate:.0%} below "
f"threshold {regression_threshold:.0%}",
diff,
)
# Approve with notes
note = diff.recommendation
if diff.regressed_tests:
note += (
f" ({len(diff.regressed_tests)} non-critical regressions "
f"accepted per threshold)"
)
return True, note, diff
# ---------------------------------------------------------------------------
# Demo
# ---------------------------------------------------------------------------
if __name__ == "__main__":
# Define a regression test suite
suite = [
RegressionTest(
test_id="happy_001",
description="Basic question answering",
user_message="What is the capital of France?",
expected_properties=[
"Response mentions Paris",
"Response is concise and direct",
],
forbidden_properties=[
"Response asks for clarification",
"Response refuses to answer",
],
source=TestSource.HAPPY_PATH,
severity=Severity.CRITICAL,
),
RegressionTest(
test_id="edge_001",
description="Empty input handling",
user_message=" ",
expected_properties=[
"Response asks for clarification or input",
],
forbidden_properties=[
"Response makes up a question to answer",
"Response throws an error message",
],
source=TestSource.EDGE_CASE,
severity=Severity.HIGH,
),
RegressionTest(
test_id="reg_001",
description="Regression: multilingual input must not break capitalization",
user_message="Como funciona OAuth? Give me an example in Python.",
expected_properties=[
"Response provides a Python code example",
"Response answers the OAuth question",
],
forbidden_properties=[
"Response only answers in Spanish ignoring the English part",
"Response refuses due to language mixing",
],
source=TestSource.PRODUCTION_FAILURE,
severity=Severity.HIGH,
),
]
old_prompt = "You are a helpful AI assistant. Answer questions clearly and concisely."
new_prompt = (
"You are a helpful AI assistant. Answer questions clearly and concisely. "
"You support all languages and mixed-language queries."
)
# Estimate cost before running
runner = PromptTestRunner()
cost = runner.estimate_cost(suite, model="claude-opus-4-6")
print(f"Estimated cost: ${cost['usd']:.4f}")
print(f"API calls: {cost['subject_api_calls']} subject + "
f"{cost['judge_api_calls']} judge\n")
# Run differential test
tester = DifferentialTester()
diff = tester.compare(old_prompt, new_prompt, suite, n_runs=1)
print(f"Improved: {len(diff.improved_tests)} tests")
print(f"Regressed: {len(diff.regressed_tests)} tests")
print(f"Unchanged: {len(diff.unchanged_tests)} tests")
print(f"Recommendation: {diff.recommendation}")
if diff.critical_regressions:
print(f"CRITICAL regressions: {diff.critical_regressions}")
Architecture Diagrams
Regression Test Lifecycle
Test Coverage Pyramid
Differential Testing: Baseline vs. Candidate
Production Notes
:::danger Never Skip the CRITICAL Suite The CRITICAL test suite should run on every prompt change, no exceptions. These tests represent the behaviors that, if broken, would immediately harm users or business. Budget and time constraints should never justify skipping them - if the suite takes 5 minutes and that is too long, the solution is to reduce the number of CRITICAL tests (only truly critical behaviors should be marked CRITICAL), not to skip the suite. :::
:::warning Flaky Tests Are A Signal, Not A Problem To Ignore A test with a 40-60% pass rate is telling you something important: the behavior it is testing is under-constrained by the current system prompt. The LLM is producing the expected output some of the time based on the probabilistic structure of the prompt, not because the prompt reliably produces it. The fix is to add explicit constraints to the system prompt that make the behavior deterministic, then verify the test stabilizes above 70% pass rate. :::
:::tip Behavioral Properties Over String Matching Regression tests should specify behavioral properties ("response provides a code example", "response does not claim to be human") not exact output strings. String matching makes tests brittle - any rephrasing breaks them. Property-based evaluation using an LLM judge is more robust and more meaningful. The only exception: outputs with fixed formats (JSON schemas, specific codes) where exact matching is appropriate. :::
:::note Test Cache Invalidation The regression cache key should include both the test ID and a hash of the system prompt. When you change the prompt, cached results from the old prompt are automatically invalidated - you always run fresh tests on a new prompt version. Do not include model version in the cache key unless you are specifically comparing model versions; caching across model versions will mask model-specific regressions. :::
Interview Questions and Answers
Q1: Why is regression testing for prompts harder than regression testing for traditional software?
Traditional software regression testing is deterministic: the same input always produces the same output, so a passing test will always pass until the code changes. Prompt regression testing is probabilistic: the LLM may produce different outputs across runs even with the same prompt and input. This requires running each test multiple times and using a pass-rate threshold rather than a binary pass/fail, detecting flaky tests separately from failing tests, and using an LLM judge for behavioral property evaluation rather than exact string matching. The non-determinism also means that a "fix" to a failing test may not be a true fix - it may just be lucky sampling - requiring statistical verification that the pass rate has genuinely improved above threshold.
Q2: How do you design regression test properties so they are stable across prompt changes?
Good regression test properties are behavioral, not syntactic. Instead of "response contains the word 'Paris'", write "response correctly identifies Paris as the capital of France". The latter will pass for "The capital is Paris", "Paris is France's capital", and "La capitale est Paris" equally, while only the former passes for the first phrasing. Similarly, instead of "response has fewer than 100 words", write "response is concise and does not include unnecessary preamble". Properties should also be scoped to specific behaviors rather than overall quality: "response provides a code example" is stable, "response is high quality" is not. Finally, forbidden properties are often more stable than expected ones - it is easier to specify what should never happen ("response claims to be human", "response provides medical diagnosis") than to enumerate everything that should happen.
Q3: What is a flaky test and how do you distinguish a flaky test from a genuinely fixed regression?
A flaky test is one that passes and fails non-deterministically on the same prompt version - its behavior is not determined by the prompt content but by the LLM's stochastic sampling. You detect it by running the test many times (10+) and computing the pass rate. If the pass rate is between 30-70%, the test is flaky. A genuinely fixed regression should have a pass rate above 70% consistently. The distinction matters for test suite maintenance: a flaky test should trigger a prompt improvement (adding constraints to make the behavior deterministic), not be marked as fixed. If a test was failing at 0% and is now at 55%, the prompt change has partially addressed the issue but has not fully resolved it.
Q4: How do you measure test coverage for a prompt regression suite?
Coverage for prompt regression testing is measured against production traffic distribution, not code line coverage. The goal is to ensure your test suite represents the same query types, topics, and difficulty levels as real user traffic. Measure coverage by: computing the distribution of production queries by topic and type (using clustering or topic classification), computing the same distribution for your test suite, and comparing the two. Coverage gaps are production topics that represent more than 5% of traffic but have fewer than a proportional number of test cases. You can also measure coverage of failure modes: for each known category of system failure, does the test suite include at least one regression test? A suite that is well-distributed across query types but has no adversarial cases has low adversarial coverage regardless of its size.
Q5: How do you structure a CI gate for prompt changes in a team with 5+ engineers shipping changes daily?
The gate should be tiered by speed and scope. For pull requests (PR stage): run CRITICAL tests only, in parallel, targeting under 5 minutes. Any CRITICAL failure blocks the PR merge. For merges to the main branch (merge stage): run CRITICAL + HIGH tests, targeting under 30 minutes. A failing CRITICAL test blocks deployment; a failing HIGH test generates an alert for human review. For production deployment (deploy gate): run the full suite including MEDIUM and LOW severity tests, estimate cost, compare pass rate to a threshold (typically 95%). The gate also runs a differential comparison against the current production prompt to show the specific tests that improved or regressed. Automated differential output posted as a PR comment gives reviewers precise information about what changed behaviorally, not just "evaluation score changed from X to Y".
Q6: How do you handle the cost of running a large regression suite frequently?
Cost management for regression suites is a real constraint - a suite of 200 tests run 3 times each with a judge evaluation amounts to 600+ API calls per suite run. Strategies: (1) Use caching - tests that have not changed (same test_id and prompt_hash) return cached results without an API call, cutting costs dramatically when only a few tests are affected. (2) Use tiers - run CRITICAL tests on every PR but full suite only on merge or deploy. (3) Use a cheap model for judge evaluation - claude-haiku at a fraction of claude-opus-4-6's cost for property checking, reserving the expensive model for the subject system under test. (4) Run MEDIUM/LOW tests on a 20% random sample per PR and run the full set weekly. (5) Budget-constrained runs that prioritize CRITICAL tests and use remaining budget for MEDIUM.