LLM as Agent Judge

The Economics of Agent Evaluation

Human evaluation costs $10 per evaluation and takes 5 minutes. An LLM judge evaluates 1000 runs overnight for approximately$ 50. Not perfect - but when calibrated correctly, it correlates strongly with human judgment.

This is the deal at the center of the LLM-as-judge paradigm: you trade some evaluation accuracy for a 1000× improvement in evaluation throughput and a 200× reduction in cost. Done carelessly, you get noise instead of signal. Done well, you get a scalable quality measurement system that actually improves your agent over time.

The field learned this the hard way. Early LLM judges were applied naively - an LLM asked "is this output good?" on a 1-5 scale. The scores looked reasonable until someone noticed the judge strongly preferred longer responses, always preferred the first option in pairwise comparisons, and consistently rated its own outputs as 5/5. Without bias detection and mitigation, LLM judgment is unreliable. With careful design, it becomes a production engineering tool.

:::tip 🎮 Interactive Playground Visualize this concept: Try the LLM as Judge demo on the EngineersOfAI Playground - no code required. :::

Why This Exists

The Human Evaluation Bottleneck

Imagine you are iterating on an agent: tweaking system prompts, adjusting tool descriptions, testing new model versions. Each iteration needs evaluation. You have 200 representative test cases. A human reviewer takes 5 minutes per case: 1000 minutes = 17 hours per iteration. You iterate 5 times per week. That is 85 hours of human evaluation per week. For one engineer, that is more than two full work weeks.

At this pace, you can afford maybe one iteration per week, properly evaluated. Everything else is shipped on intuition.

LLM-as-judge changes this. At 1000 cases per hour, you can evaluate every iteration, including experimental ones. You catch regressions before they ship. You compare alternatives quantitatively. You move from gut-feel engineering to data-driven iteration.

Historical Context

LLM-as-judge emerged from the MT-Bench paper (Zheng et al., 2023) from UC Berkeley, which introduced GPT-4 as a judge for chatbot evaluation and showed it correlated with human preference at 80%+ agreement. This was the proof-of-concept. Since then, it has become standard practice at major AI labs for evaluating both models and agents.

The critical insight from MT-Bench: the judge must be more capable than the system being judged. A weaker model cannot reliably identify errors made by a stronger model. If you are evaluating a Claude Opus agent, you need at least Claude Opus (or equivalent) as the judge.

What LLM Judges Excel At vs What They Fail At

LLM Judges Excel	LLM Judges Fail
Holistic quality assessment	Factual verification
Coherence and logical flow	Code correctness
Helpfulness and relevance	Numerical accuracy
Coverage of key points	Safety edge cases
Clarity and communication style	Domain-specific correctness
Identifying obvious hallucinations	Subtle reasoning errors
Ranking relative quality	Absolute calibration

The core limitation: LLM judges share the same failure modes as LLMs generally. If the model being evaluated hallucinates a fact convincingly, the judge is likely to accept it. If the model makes a subtle mathematical error, the judge may miss it. If the model produces unsafe content that is superficially polished, the judge may rate it highly.

Use LLM judges for general quality assessment. Use specialized checks (code execution, fact verification APIs, safety classifiers) for domain-specific correctness.

Judge Prompt Designs

1. Rubric-Based Scoring

The most reliable approach: define explicit criteria with clear definitions for each score point.

You are evaluating an AI assistant's response to a user query.
Score the response on the following dimensions:

ACCURACY (1-5):
1 - Contains significant factual errors
2 - Contains minor factual errors
3 - Mostly accurate with small uncertainties
4 - Accurate, well-supported
5 - Accurate and verifiably correct

HELPFULNESS (1-5):
1 - Fails to address the user's need
2 - Partially addresses the need
3 - Addresses the main need but misses details
4 - Comprehensively helpful
5 - Exceeds expectations, anticipates follow-ups

CLARITY (1-5):
1 - Confusing or poorly organized
2 - Somewhat unclear
3 - Clear for most readers
4 - Very clear and well-organized
5 - Exceptionally clear, perfect organization

Provide scores and a one-sentence justification for each dimension.
Format: ACCURACY: [score]/5 - [justification]

Rubric-based scoring gives you multi-dimensional quality signals, detailed justifications, and reproducible criteria. The tradeoff: rubric design requires significant iteration, and rubrics can fail to capture nuances not anticipated in the criteria.

2. Pairwise Comparison

Given two responses A and B, ask: which is better?

Pairwise comparison produces stronger signal than absolute scoring because it sidesteps calibration difficulties - a judge doesn't need to know what a "4/5" means, it only needs to know which is better.

You are comparing two AI responses to the same user query.

Query: [user's question]

Response A:
[response A]

Response B:
[response B]

Which response is better? Consider:
- Accuracy and correctness
- Helpfulness to the user
- Clarity and organization
- Completeness

Answer: "A" if Response A is clearly better, "B" if Response B is clearly better,
or "TIE" if they are approximately equal.
Reasoning: [brief explanation]

Pairwise comparison is ideal for comparing two agent versions on the same tasks. Run A vs B on 50–100 shared tasks. Count wins, losses, ties. A win rate above 55% for the challenger suggests genuine improvement.

3. Reference-Based Scoring

When a gold reference exists, compare the agent output against it:

You are evaluating whether an AI agent's response matches a reference answer.

Question: [question]
Reference answer: [gold answer]
Agent's answer: [agent's answer]

Score semantic equivalence (1-5):
1 - Contradicts or completely misses the reference
2 - Partially matches, major omissions
3 - Matches main points, misses details
4 - Strong match with minor differences
5 - Equivalent or better than reference

Note: The agent's response may be correct even if phrased differently.
Score 5 if the meaning is equivalent, not only if words match.

4. Reference-Free Scoring

No gold reference exists. Evaluate purely on quality properties:

You are evaluating an AI agent's response to a user query.
There is no reference answer - evaluate on absolute quality.

Query: [query]
Response: [response]

Does this response:
1. Answer the actual question asked? (Y/N)
2. Provide accurate information? (Y/N/CANNOT_VERIFY)
3. Give sufficient detail for the user's apparent need? (Y/N)
4. Avoid harmful, misleading, or inappropriate content? (Y/N)

Overall quality (1-5): [score]
Critical issues: [any blocking problems]

Bias Types and Mitigations

Position Bias: The Most Dangerous

In pairwise comparison, LLM judges systematically prefer the response they see first. This bias can be 5–15 percentage points depending on the judge model. Left unmitigated, it will give false wins to whichever version you put in position A.

Mitigation: Run every pairwise comparison twice - once with A first, once with B first. Only count it as a win for A if A wins in both orderings. If the result flips with ordering, record it as a tie.

Verbosity Bias: Sneaky and Common

Longer responses look more thorough. LLM judges are susceptible to this illusion - a response with 500 words can outscore a response with 200 words even when the shorter response is actually better.

Mitigation: Add an explicit instruction: "Ignore response length. A concise, accurate answer is better than a verbose one that adds padding. Score solely on quality of information, not quantity."

Self-Preference: Structural Problem

When using the same model family as judge that you are evaluating (e.g., Claude judging Claude), the judge tends to prefer outputs in its own style. This inflates scores for agents using the same model family.

Mitigation: Use a different model family as judge (e.g., GPT-4o judging Claude outputs). For highest reliability, use an ensemble of judges from different families.

Calibration Methodology

Calibration answers: how well does my LLM judge agree with human judgment?

Step 1: Collect a Calibration Set

Sample 100–200 agent outputs that span the full quality distribution - including clearly good, clearly bad, and ambiguous cases. Have human annotators rate each on your rubric dimensions.

Step 2: Run Your LLM Judge

Apply the LLM judge to the same set of outputs, using the same rubric dimensions.

Step 3: Measure Agreement

from scipy.stats import spearmanr, pearsonr
from sklearn.metrics import cohen_kappa_score

def compute_calibration_metrics(human_scores, judge_scores):
    """Compute multiple agreement metrics between human and LLM judge scores."""
    assert len(human_scores) == len(judge_scores)

    # Spearman rank correlation - appropriate for ordinal scores
    spearman_r, spearman_p = spearmanr(human_scores, judge_scores)

    # Pearson correlation - appropriate for continuous scores
    pearson_r, pearson_p = pearsonr(human_scores, judge_scores)

    # Cohen's kappa - for categorical/discretized scores
    # Requires discretizing to same set of categories
    kappa = cohen_kappa_score(
        [round(s) for s in human_scores],
        [round(s) for s in judge_scores]
    )

    # Exact agreement rate
    exact_agreement = sum(
        round(h) == round(j)
        for h, j in zip(human_scores, judge_scores)
    ) / len(human_scores)

    # Within-1 agreement rate
    within_1 = sum(
        abs(round(h) - round(j)) <= 1
        for h, j in zip(human_scores, judge_scores)
    ) / len(human_scores)

    return {
        "spearman_r": round(spearman_r, 3),
        "pearson_r": round(pearson_r, 3),
        "cohen_kappa": round(kappa, 3),
        "exact_agreement": round(exact_agreement, 3),
        "within_1_agreement": round(within_1, 3),
    }

# Interpretation targets
CALIBRATION_THRESHOLDS = {
    "spearman_r": {"good": 0.75, "acceptable": 0.60},
    "cohen_kappa": {"good": 0.70, "acceptable": 0.50},
    "within_1_agreement": {"good": 0.85, "acceptable": 0.75},
}

Target calibration values:

Spearman correlation > 0.75: your judge is reliable
Cohen's kappa > 0.70: strong agreement
Within-1 agreement > 85%: excellent

Below these thresholds, the judge should not be used as a standalone signal - it needs either human co-evaluation or significant prompt improvement.

Ensemble Judges

A single judge is biased. Multiple judges with majority vote is more reliable.

The ensemble principle: if you use judges from three different model families (e.g., Claude, GPT-4o, Gemini), each has different biases. In the majority vote, individual biases cancel out. Agreement across all three is a strong signal.

def ensemble_pairwise_judgment(query, response_a, response_b, judges):
    """
    Run pairwise comparison with multiple judge models.
    Returns majority vote.
    """
    votes = {"A": 0, "B": 0, "TIE": 0}

    for judge_fn in judges:
        # Run each pair in both orderings
        result_1 = judge_fn(query, response_a, response_b)  # A first
        result_2 = judge_fn(query, response_b, response_a)  # B first (swap)

        # Only count if consistent across orderings
        if result_1 == "A" and result_2 == "B":
            votes["A"] += 1
        elif result_1 == "B" and result_2 == "A":
            votes["B"] += 1
        else:
            votes["TIE"] += 1

    # Majority vote
    winner = max(votes, key=votes.get)
    confidence = votes[winner] / sum(votes.values())
    return winner, confidence, votes

Full Python: LLM-as-Judge System

"""
Production LLM-as-judge system with rubrics, pairwise comparison,
calibration, confidence estimation, and human escalation logic.
"""

import json
import random
import time
from dataclasses import dataclass, field
from typing import Optional
import anthropic

client = anthropic.Anthropic()


# ── Rubric definitions ─────────────────────────────────────────────────────────

AGENT_EVALUATION_RUBRIC = {
    "task_completion": {
        "description": "Did the agent fully complete the requested task?",
        "criteria": {
            1: "Task not completed. Agent gave up, went off-topic, or produced irrelevant output.",
            2: "Task partially completed. Major aspects missing or incorrect.",
            3: "Task mostly completed. Minor aspects missing.",
            4: "Task fully completed with minor imperfections.",
            5: "Task fully completed, exceeds expectations.",
        }
    },
    "factual_accuracy": {
        "description": "Are the facts and claims in the response correct?",
        "criteria": {
            1: "Multiple significant factual errors.",
            2: "One significant factual error.",
            3: "Mostly accurate, minor uncertainties.",
            4: "Accurate. All verifiable claims are correct.",
            5: "Highly accurate with strong sourcing.",
        }
    },
    "reasoning_quality": {
        "description": "Is the agent's reasoning sound and logical?",
        "criteria": {
            1: "Illogical or incoherent reasoning.",
            2: "Reasoning has notable gaps or errors.",
            3: "Reasonable but not thorough.",
            4: "Sound, well-structured reasoning.",
            5: "Excellent reasoning, anticipates edge cases.",
        }
    },
    "response_quality": {
        "description": "Is the response well-organized, clear, and appropriately formatted?",
        "criteria": {
            1: "Poorly organized, confusing.",
            2: "Some clarity issues.",
            3: "Acceptable organization.",
            4: "Well-organized and clear.",
            5: "Exceptionally clear and well-formatted.",
        }
    },
}


# ── Data models ────────────────────────────────────────────────────────────────

@dataclass
class RubricScore:
    dimension: str
    score: int           # 1-5
    justification: str
    confidence: float    # 0-1, judge's self-assessed confidence


@dataclass
class JudgeResult:
    query: str
    response: str
    rubric_scores: list[RubricScore]
    composite_score: float         # weighted average across dimensions
    confidence: float              # overall judge confidence
    escalate_to_human: bool        # True if confidence is too low
    judge_reasoning: str
    judge_model: str
    duration_ms: float


@dataclass
class PairwiseResult:
    query: str
    response_a: str
    response_b: str
    winner: str                    # "A", "B", or "TIE"
    confidence: float
    a_first_result: str
    b_first_result: str
    consistent: bool               # True if same result regardless of order
    reasoning: str


# ── Judge implementation ───────────────────────────────────────────────────────

class LLMJudge:
    """
    Production LLM-as-judge with rubric scoring, pairwise comparison,
    bias mitigation, and escalation logic.
    """

    ESCALATION_THRESHOLD = 0.55   # Escalate if judge confidence below this

    def __init__(
        self,
        judge_model: str = "claude-opus-4-6",
        rubric: dict = None,
        dimension_weights: dict = None,
    ):
        self.judge_model = judge_model
        self.rubric = rubric or AGENT_EVALUATION_RUBRIC
        self.dimension_weights = dimension_weights or {
            "task_completion": 0.40,
            "factual_accuracy": 0.30,
            "reasoning_quality": 0.20,
            "response_quality": 0.10,
        }

    def evaluate_rubric(
        self,
        query: str,
        response: str,
        context: Optional[str] = None,
    ) -> JudgeResult:
        """
        Score a response on all rubric dimensions.
        """
        rubric_text = self._format_rubric()
        context_text = f"\nAdditional context: {context}" if context else ""

        prompt = f"""You are a careful, objective evaluator of AI agent responses.

Evaluate the following response on each dimension of the rubric.
Be precise and honest. Do not be lenient - your scores calibrate our systems.

User query: {query}{context_text}

Agent response:
{response}

{rubric_text}

For each dimension, provide:
1. A score (1-5)
2. A one-sentence justification
3. Your confidence in the score (0.0-1.0)

Respond in this exact JSON format:
{{
  "scores": {{
    "task_completion": {{"score": X, "justification": "...", "confidence": 0.X}},
    "factual_accuracy": {{"score": X, "justification": "...", "confidence": 0.X}},
    "reasoning_quality": {{"score": X, "justification": "...", "confidence": 0.X}},
    "response_quality": {{"score": X, "justification": "...", "confidence": 0.X}}
  }},
  "overall_reasoning": "Brief overall assessment"
}}

Important: Ignore response length. A concise, accurate answer is better than a verbose one.
"""

        t0 = time.time()
        api_response = client.messages.create(
            model=self.judge_model,
            max_tokens=1500,
            messages=[{"role": "user", "content": prompt}],
        )
        duration_ms = (time.time() - t0) * 1000

        result_text = api_response.content[0].text
        parsed = self._parse_rubric_response(result_text)

        rubric_scores = []
        for dim, data in parsed.get("scores", {}).items():
            rubric_scores.append(RubricScore(
                dimension=dim,
                score=int(data.get("score", 3)),
                justification=data.get("justification", ""),
                confidence=float(data.get("confidence", 0.5)),
            ))

        # Compute composite score
        composite = self._compute_composite(rubric_scores)
        avg_confidence = (
            sum(s.confidence for s in rubric_scores) / len(rubric_scores)
            if rubric_scores else 0.5
        )

        return JudgeResult(
            query=query,
            response=response,
            rubric_scores=rubric_scores,
            composite_score=composite,
            confidence=avg_confidence,
            escalate_to_human=avg_confidence < self.ESCALATION_THRESHOLD,
            judge_reasoning=parsed.get("overall_reasoning", ""),
            judge_model=self.judge_model,
            duration_ms=duration_ms,
        )

    def evaluate_pairwise(
        self,
        query: str,
        response_a: str,
        response_b: str,
    ) -> PairwiseResult:
        """
        Compare two responses. Runs in both A-first and B-first orders
        to mitigate position bias.
        """
        # Order 1: A first
        result_a_first = self._single_pairwise(query, response_a, response_b, "A", "B")

        # Order 2: B first (swap positions)
        result_b_first = self._single_pairwise(query, response_b, response_a, "B", "A")
        # Translate back to A/B labels (in this call, "A" in position was response_b)
        if result_b_first == "A":
            result_b_first_translated = "B"
        elif result_b_first == "B":
            result_b_first_translated = "A"
        else:
            result_b_first_translated = "TIE"

        # Determine final winner
        consistent = result_a_first == result_b_first_translated

        if result_a_first == result_b_first_translated:
            winner = result_a_first
            confidence = 0.90 if winner != "TIE" else 0.80
        else:
            # Inconsistent: position bias likely, call it a TIE
            winner = "TIE"
            confidence = 0.50

        reasoning = self._generate_pairwise_reasoning(
            query, response_a, response_b, winner
        )

        return PairwiseResult(
            query=query,
            response_a=response_a,
            response_b=response_b,
            winner=winner,
            confidence=confidence,
            a_first_result=result_a_first,
            b_first_result=result_b_first_translated,
            consistent=consistent,
            reasoning=reasoning,
        )

    def _single_pairwise(
        self,
        query: str,
        response_first: str,
        response_second: str,
        label_first: str,
        label_second: str,
    ) -> str:
        """Single pairwise comparison with specified ordering."""
        prompt = f"""Compare these two AI responses to the same query.

Query: {query}

Response {label_first}:
{response_first}

Response {label_second}:
{response_second}

Which response is better? Consider: accuracy, helpfulness, completeness, and clarity.
Ignore length - a concise accurate answer beats a verbose one.

Answer with exactly one of: "{label_first}", "{label_second}", or "TIE"
Then one sentence of reasoning.

Format:
WINNER: [choice]
REASON: [one sentence]
"""

        api_response = client.messages.create(
            model=self.judge_model,
            max_tokens=200,
            messages=[{"role": "user", "content": prompt}],
        )

        text = api_response.content[0].text.strip()

        if f"WINNER: {label_first}" in text:
            return label_first
        elif f"WINNER: {label_second}" in text:
            return label_second
        else:
            return "TIE"

    def _generate_pairwise_reasoning(
        self, query: str, response_a: str, response_b: str, winner: str
    ) -> str:
        """Generate explanation for pairwise result."""
        if winner == "TIE":
            return "Responses are approximately equivalent in quality."

        better = response_a if winner == "A" else response_b
        worse = response_b if winner == "A" else response_a

        prompt = f"""Response {winner} was determined to be better.

Query: {query}
Response A: {response_a[:300]}...
Response B: {response_b[:300]}...
Winner: {winner}

In one sentence, explain why {winner} is better."""

        api_response = client.messages.create(
            model=self.judge_model,
            max_tokens=150,
            messages=[{"role": "user", "content": prompt}],
        )
        return api_response.content[0].text.strip()

    def _format_rubric(self) -> str:
        lines = ["EVALUATION RUBRIC:"]
        for dim, spec in self.rubric.items():
            lines.append(f"\n{dim.upper()}: {spec['description']}")
            for score, desc in spec["criteria"].items():
                lines.append(f"  {score}: {desc}")
        return "\n".join(lines)

    def _parse_rubric_response(self, text: str) -> dict:
        """Parse JSON response from judge, handling malformed output."""
        # Try to extract JSON from the response
        try:
            # Find JSON block
            start = text.find("{")
            end = text.rfind("}") + 1
            if start >= 0 and end > start:
                return json.loads(text[start:end])
        except json.JSONDecodeError:
            pass

        # Fallback: default scores
        return {
            "scores": {
                dim: {"score": 3, "justification": "Parse error", "confidence": 0.3}
                for dim in self.rubric
            },
            "overall_reasoning": "Failed to parse judge response",
        }

    def _compute_composite(self, rubric_scores: list[RubricScore]) -> float:
        """Compute weighted composite score (1-5 normalized to 0-1)."""
        if not rubric_scores:
            return 0.5

        total_weight = 0
        weighted_sum = 0

        for score in rubric_scores:
            weight = self.dimension_weights.get(score.dimension, 0.25)
            weighted_sum += weight * score.score
            total_weight += weight

        raw_score = weighted_sum / total_weight if total_weight > 0 else 3.0
        return (raw_score - 1) / 4.0  # Normalize to [0, 1]


# ── Judge pipeline ─────────────────────────────────────────────────────────────

@dataclass
class EvaluationPipelineResult:
    query: str
    agent_response: str
    judge_result: JudgeResult
    escalated: bool
    human_review_required: bool
    action: str   # "accept", "flag", "escalate"


class JudgePipeline:
    """
    Full evaluation pipeline: judge → confidence check → escalation.
    """

    def __init__(
        self,
        judge: LLMJudge,
        quality_threshold: float = 0.60,   # Min composite score to accept
        confidence_threshold: float = 0.55, # Min confidence before human review
        alert_fn=None,                       # Optional: fn(query, result) to alert on bad outputs
    ):
        self.judge = judge
        self.quality_threshold = quality_threshold
        self.confidence_threshold = confidence_threshold
        self.alert_fn = alert_fn

    def evaluate(self, query: str, agent_response: str) -> EvaluationPipelineResult:
        """Run the full evaluation pipeline."""
        judge_result = self.judge.evaluate_rubric(query, agent_response)

        # Determine action
        low_quality = judge_result.composite_score < self.quality_threshold
        low_confidence = judge_result.confidence < self.confidence_threshold

        if low_quality and not low_confidence:
            action = "flag"           # Judge is confident this is bad
        elif low_confidence:
            action = "escalate"       # Judge is uncertain, need human
        else:
            action = "accept"

        result = EvaluationPipelineResult(
            query=query,
            agent_response=agent_response,
            judge_result=judge_result,
            escalated=action == "escalate",
            human_review_required=action in ("escalate", "flag"),
            action=action,
        )

        # Alert on bad outputs
        if action == "flag" and self.alert_fn:
            self.alert_fn(query, result)

        return result

    def batch_evaluate(
        self,
        queries_and_responses: list[tuple[str, str]],
    ) -> dict:
        """Evaluate a batch and return summary statistics."""
        results = []
        for query, response in queries_and_responses:
            result = self.evaluate(query, response)
            results.append(result)

        accepted = [r for r in results if r.action == "accept"]
        flagged = [r for r in results if r.action == "flag"]
        escalated = [r for r in results if r.action == "escalate"]

        scores = [r.judge_result.composite_score for r in results]

        return {
            "total": len(results),
            "accepted": len(accepted),
            "flagged": len(flagged),
            "escalated": len(escalated),
            "accept_rate": len(accepted) / len(results),
            "mean_score": sum(scores) / len(scores),
            "min_score": min(scores),
            "max_score": max(scores),
            "results": results,
        }


# ── Demo ───────────────────────────────────────────────────────────────────────

def demo():
    judge = LLMJudge(judge_model="claude-opus-4-6")
    pipeline = JudgePipeline(judge, quality_threshold=0.60, confidence_threshold=0.55)

    test_cases = [
        (
            "What is the capital of France?",
            "The capital of France is Paris. It has been the capital since the 10th century."
        ),
        (
            "Explain gradient descent in machine learning.",
            "Gradient descent is an optimization algorithm. It works.",
        ),
    ]

    print("Running LLM-as-judge evaluation pipeline...\n")

    for query, response in test_cases:
        result = pipeline.evaluate(query, response)

        print(f"Query: {query[:60]}...")
        print(f"Action: {result.action.upper()}")
        print(f"Composite score: {result.judge_result.composite_score:.3f}")
        print(f"Judge confidence: {result.judge_result.confidence:.3f}")

        for score in result.judge_result.rubric_scores:
            print(f"  {score.dimension}: {score.score}/5 - {score.justification}")

        print(f"Reasoning: {result.judge_result.judge_reasoning}")
        print()


if __name__ == "__main__":
    demo()

Production Engineering Notes

Version Your Rubrics

A rubric change is a measurement change. When you update rubric criteria, re-evaluate a holdout set from before the change. Track rubric versions alongside eval results. Never compare scores computed with different rubric versions without this.

Track Judge Model Changes

When the judge model is upgraded, re-calibrate against human ratings. A newer judge model may score systematically higher or lower. Recalibration ensures your thresholds remain meaningful.

Estimate Uncertainty From Agreement

The best way to estimate judge uncertainty is to run two independent judge calls on the same input and measure agreement. High disagreement = low confidence, escalate to human.

Cost-Quality Tradeoffs

Using the most capable judge model (e.g., Claude Opus) for every evaluation is expensive. A tiered approach works well: use a cheaper model (Claude Haiku) for initial triage, escalate low-confidence or flagged items to the more capable model, escalate the remaining uncertain items to humans.

:::danger Never Trust a Single Judge Metric LLM judge scores are not ground truth - they are one signal. A single dimension score of 4.2/5 might mean "very good" or it might mean "the judge was fooled by a confident-sounding but incorrect response." Always interpret judge scores alongside other signals: trajectory metrics, task completion rates, and - periodically - human evaluation to verify calibration. :::

:::warning Self-Preference Bias Is Larger Than You Think When using Claude to judge Claude's outputs, the preference for Claude-style responses can be 10–15 percentage points. This is enough to completely obscure real quality differences when comparing Claude agents to agents built on other models. Always use a different model family as judge, or use an ensemble that includes judges from multiple families. :::

Interview Q&A

Q: What is LLM-as-judge and when is it appropriate to use?

A: LLM-as-judge is using a capable language model to evaluate the outputs of another LLM or agent. It is appropriate when: human evaluation is too expensive or slow for the evaluation volume needed, the evaluation task requires holistic quality judgment (coherence, helpfulness, relevance) rather than objective fact-checking, and the judge model is at least as capable as the model being evaluated. It is not appropriate as the sole signal for safety-critical evaluation, factual accuracy checking, or code correctness verification - domains where LLM judges share failure modes with the systems they are evaluating.

Q: Explain position bias in LLM-as-judge and how to mitigate it.

A: Position bias is the tendency of LLM judges in pairwise comparison to prefer whichever response appears first. The bias can be 5–15 percentage points depending on the model, which is large enough to reverse the actual quality ordering. Mitigation: run every pairwise comparison in both orderings (A first, then B first). Only count a win if the same response wins in both orderings. If the result flips with ordering, record a tie - this is the appropriate outcome when position bias is detected. This doubles evaluation cost but produces unbiased results.

Q: How do you calibrate an LLM judge? What agreement level is acceptable?

A: Calibration involves collecting 100–200 agent outputs spanning the quality spectrum, having human annotators rate them on the same rubric, running the LLM judge on the same set, then measuring agreement using Spearman rank correlation (for ordered scores), Cohen's kappa (for categorical agreement), and within-1 agreement rate (fraction of cases where judge and human differ by at most 1 point). Acceptable targets: Spearman r > 0.75, Cohen's kappa > 0.70, within-1 agreement > 85%. Below these, the judge needs significant prompt improvement before being used as a standalone signal. Re-calibrate whenever the judge model changes or the rubric is updated.

Q: What should you escalate to human evaluation instead of relying on LLM judgment?

A: Three escalation triggers: low judge confidence (when the judge's self-assessed confidence falls below a threshold, typically 0.55, because uncertain judge scores are noisy and unreliable), novel or unusual outputs (content that falls outside the judge's calibration distribution - surprising claims, unusual formats, complex technical content), and safety-relevant content (any output that might violate safety guidelines - LLM judges are not reliable safety classifiers and should never be the final gate for safety). Additionally, run periodic human review (10% sample of accepted outputs) to detect when judge calibration has drifted.

Q: How would you design an ensemble judge system for higher reliability?

A: An ensemble uses multiple judge models from different families (e.g., Claude, GPT-4o, Gemini) with a voting or averaging mechanism. For rubric scoring: compute scores from all judges, average the scores, report disagreement as uncertainty. For pairwise comparison: each judge votes A, B, or TIE; use majority vote as the winner; report confidence as the vote fraction. The key benefit: biases specific to one model family (position bias magnitude, verbosity preference, self-preference) are different across models, and majority voting reduces the impact of any individual model's bias. Three judges from different families produce results that correlate with human judgment better than any single judge alone.

The LLM Judge Integration Map

Where LLM judges fit into the broader evaluation and monitoring stack:

The key relationships: human labels calibrate the judge (upward arrow from calibration to LLM judge), judge scores identify candidates for human review (low-confidence or low-scoring outputs), and programmatic checks run on 100% of traffic without sampling. The three layers are complementary - each covers blind spots in the others.

Summary: Building a Judge That You Can Trust

The LLM-as-judge pattern is powerful but earns trust slowly. Here is the sequence that separates a reliable judge from one that produces misleading scores:

Define evaluation criteria precisely. Vague rubrics produce vague scores. Spend time on the rubric before running a single evaluation.
Collect human labels first. Build a calibration set of 100+ human-labeled examples before deploying any automated judge.
Validate, then deploy. If Pearson correlation with human labels is below 0.7, fix the rubric or switch judge models before using it for any decision.
Apply bias mitigations by default. Swap debiasing for pairwise, explicit conciseness criteria for rubric scoring, multi-family ensembles for cross-model comparisons.
Monitor judge stability. After every model update (judge model or agent model), re-run calibration. Score drift from model changes is common and silent.
Close the human loop. Periodically review 10% of automatically accepted outputs with human annotators. This is how you catch calibration drift before it compounds.

A judge that follows these steps is not infallible - but it is trustworthy enough to drive real decisions: shipping or blocking agent versions, detecting production regressions, and identifying where to invest improvement effort. That is the standard it needs to meet.

Quick Reference: LLM Judge Checklist

Before deploying any LLM judge to production, verify:

This checklist operationalizes the principles in this lesson. A judge that passes all ten items is ready for production use as a quality signal. A judge that fails any item has a specific, fixable gap. Start with the checklist before running your first production evaluation.

The items with the highest leverage are calibration (items 3-4) and human review loop (item 8). Teams that skip calibration ship judges that measure the wrong things with high confidence. Teams that skip the human review loop deploy judges that drift undetected as user behavior and model characteristics evolve. The investment in these two items pays continuous dividends for as long as the judge is in use.

Continue to the next lesson - Human Evaluation for Agents - for the ground truth layer that makes LLM judges trustworthy in the first place. LLM judges are powerful only to the extent that they are calibrated against human judgment, and human evaluation is how that calibration happens.

Cost Reference Table

Understanding the economics of LLM judging helps teams make informed decisions about evaluation frequency and sample size:

Evaluation Type	Cost per Item	Items per Day	Daily Cost
Rubric scoring (Haiku judge)	~$0.002	5,000	~$10
Rubric scoring (Opus judge)	~$0.05	5,000	~$250
Pairwise + swap debiasing (Haiku)	~$0.005	2,500	~$12.50
Pairwise + swap debiasing (Opus)	~$0.10	2,500	~$250
Human annotation (crowdwork)	~$1.00	500	~$500
Human annotation (expert)	~$25.00	20	~$500

The economics make the tiered approach compelling: use a fast, cheap judge (Haiku) for high-volume triage, escalate low-confidence items to a capable judge (Opus), and escalate the remaining uncertain items to humans. This combination covers 100% of production volume at a cost that scales with actual quality uncertainty rather than with total volume.

Prices are approximate and based on early 2025 API pricing. Check current pricing at console.anthropic.com before planning production evaluation budgets.

The Economics of Agent Evaluation​

Why This Exists​

The Human Evaluation Bottleneck​

Historical Context​

What LLM Judges Excel At vs What They Fail At​

Judge Prompt Designs​

1. Rubric-Based Scoring​

2. Pairwise Comparison​

3. Reference-Based Scoring​

4. Reference-Free Scoring​

Bias Types and Mitigations​

Position Bias: The Most Dangerous​

Verbosity Bias: Sneaky and Common​

Self-Preference: Structural Problem​

Calibration Methodology​

Step 1: Collect a Calibration Set​

Step 2: Run Your LLM Judge​

Step 3: Measure Agreement​

Ensemble Judges​

Full Python: LLM-as-Judge System​

Production Engineering Notes​

Version Your Rubrics​

Track Judge Model Changes​

Estimate Uncertainty From Agreement​

Cost-Quality Tradeoffs​

Interview Q&A​

The LLM Judge Integration Map​

Summary: Building a Judge That You Can Trust​

Further Reading​

Quick Reference: LLM Judge Checklist​

Cost Reference Table​