Benchmarks: MMLU, HumanEval, and HELM

The Leaderboard Inflation

In March 2023, a major AI lab released a model that scored 75% on MMLU - a significant improvement over the previous state-of-the-art. The press release announced it as a milestone in AI capability. Bloggers wrote about the model approaching human-expert performance in knowledge and reasoning.

Two months later, researchers at a different institution discovered something uncomfortable: large portions of the MMLU test set appeared verbatim in the model's training data. The model had not achieved superior knowledge reasoning - it had partially memorized the test questions. The score was inflated by an unknown amount. There was no malice; the lab had no easy way to audit what was in their training crawl. But the number was wrong, and it had influenced decisions, hiring, and investment.

This is not an isolated incident. By 2024, researchers studying major public benchmarks found that for any model trained on a large internet crawl, meaningful contamination of public benchmarks is nearly impossible to rule out. The benchmark ecosystem that the field depends on for progress measurement has a systemic validity crisis.

Understanding this crisis - what each benchmark actually measures, how contamination happens, and how to evaluate models fairly - is essential for anyone who makes decisions based on benchmark scores.

Why This Exists - The Need for Standardized Capability Measurement

Before standardized benchmarks, comparing LLMs meant running custom evaluations on custom datasets, which were non-reproducible, non-comparable, and often cherry-picked to favor one's own model.

Benchmarks solve the coordination problem: if every lab evaluates on the same held-out questions with the same scoring methodology, you can compare models fairly. This is the theory. The practice has been messier.

The benchmark ecosystem emerged from NLP tasks - translation, sentiment analysis, NER - where ground truth was clear. As models became general-purpose, the benchmarks needed to scale too. MMLU represents one approach (breadth of knowledge across subjects), HumanEval represents another (functional correctness for code), and HELM represents a third (multi-metric, multi-scenario holistic view).

The Benchmark Landscape

MMLU: Massive Multitask Language Understanding

What It Is

MMLU (Hendrycks et al., 2020) is a multiple-choice benchmark covering 57 academic subjects from elementary to professional level:

• STEM: mathematics, physics, chemistry, biology, computer science, engineering
• Social sciences: economics, psychology, sociology, political science
• Humanities: history, philosophy, law, ethics
• Other: medicine, nursing, business, finance

Each question has 4 answer choices. The benchmark contains 15,908 questions total, with a validation set and test set. Evaluation is typically done in 5-shot setting (5 examples in context before the question).

What It Actually Measures

MMLU measures breadth of declarative knowledge across a wide range of academic subjects. A high MMLU score means the model has absorbed large amounts of factual knowledge from its training data. It does not measure:

• Reasoning ability (the questions are mostly fact-retrieval, not reasoning chains)
• Instruction following
• Applied problem-solving
• Knowledge currency (all questions are static)

Score Progression

Model	MMLU (5-shot)
GPT-3 (175B, 2020)	43.9%
Gopher (280B, 2021)	60.0%
PaLM (540B, 2022)	69.3%
GPT-4 (2023)	86.4%
Claude 3 Opus (2024)	88.7%
Human expert average	~89.8%
Gemini Ultra (2024)	90.0%

Why MMLU Is Saturating

By 2024, frontier models are approaching and exceeding average human expert performance on MMLU. This is both impressive and misleading. The saturation means MMLU can no longer reliably differentiate between top models - a 0.5 percentage point difference between models is within the margin of error.

Saturation happens for several reasons:

1. Models have absorbed enough training data that most MMLU questions are trivially predictable
2. The questions are multiple-choice - a structured format that models have seen extensively
3. Many questions have leaked into training data through benchmark compilation sites

The field response: develop harder benchmarks (MMLU-Pro with more complex questions, GPQA for expert-level science), private benchmarks not on the internet, and dynamic benchmarks (new questions regularly added).

HumanEval: Code Generation Evaluation

What It Is

HumanEval (Chen et al., 2021, Codex paper from OpenAI) is a code generation benchmark consisting of 164 hand-written Python programming problems. Each problem includes:

• A function signature
• A docstring describing what the function should do
• A set of unit tests that the generated function must pass

The model must generate a function body that passes all unit tests.

The pass@k Metric

HumanEval uses pass@k: the probability that at least one of k randomly sampled completions passes all unit tests.

$pass@k = \mathbb{E}\left[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$

where $n$ is the total number of samples generated, and $c$ is the number that pass the tests.

In practice, you generate $n=200$ samples per problem (because sampling is cheap) and compute an unbiased estimate of pass@1, pass@10, pass@100.

Interpretation:

• pass@1: if you take one sample, what is the probability it is correct?
• pass@10: if you take 10 samples and pick the best, what is the probability you find a correct one?
• pass@100: if you take 100 samples, what is the probability of finding a correct one?

Score Progression

Model	pass@1	pass@100
GPT-3 (codex-12B)	28.8%	72.3%
Codex (code-davinci-002)	47.0%	78.2%
GPT-4 (2023)	~85.0%	-
Claude 3.5 Sonnet (2024)	~92%	-

SWE-Bench: The Harder Alternative

HumanEval problems are clean, isolated, and synthetic. Real software engineering involves:

• Understanding a large existing codebase
• Interpreting vague natural language issue descriptions
• Making targeted changes that don't break existing tests
• Navigating dependencies and interfaces

SWE-Bench (Jimenez et al., 2023) addresses this by using 2,294 real GitHub issues from 12 popular Python repositories. The model must resolve the issue, and success is measured by whether the existing test suite passes.

SWE-Bench scores are dramatically lower than HumanEval:

• GPT-4 on SWE-Bench: ~4% resolved (original, before scaffolding)
• With agent scaffolding (SWE-agent): GPT-4 ~12%, Claude 3.5 Sonnet ~50%

This gap (85% on HumanEval vs 4% on SWE-Bench for GPT-4) illustrates the difference between benchmark-tuned capability and real-world software engineering ability.

HELM: Holistic Evaluation of Language Models

What It Is

HELM (Liang et al., 2022, Stanford) is a framework, not a single benchmark. It evaluates models across multiple scenarios, each measured by multiple metrics. The goal is to reveal a complete capability profile, not a single number.

HELM includes:

• 42 evaluation scenarios covering diverse tasks and domains
• 7 metrics per scenario: accuracy, calibration, robustness, fairness, bias, toxicity, efficiency
• Standardized evaluation infrastructure to ensure reproducibility

Why Multi-Metric Matters

A model might have:

• High accuracy but poor calibration (overconfident)
• Good accuracy but high toxicity (problematic outputs)
• Strong performance on majority groups but poor performance on minorities (fairness issue)
• Excellent accuracy but only when running at very high compute cost (efficiency problem)

HELM surfaces all of these dimensions simultaneously. A model that scores 70% accuracy with 0.95 calibration score and no demographic performance gaps is genuinely better than a model that scores 75% accuracy while being poorly calibrated and showing large performance differences across demographic groups.

HELM Core Scenarios

Category	Scenarios
QA	NaturalQuestions, TriviaQA, ARC
Text Classification	SST, MNLI, HateSpeech18
Summarization	CNN/DailyMail, XSum, BoolQ
Information Retrieval	MS-MARCO
Reasoning	HellaSwag, OpenBookQA, MATH
Disinformation	TruthfulQA, NLI
Copyright	Books, News articles
Harms	BBQ (bias), WinoBias, ToxiGen

GSM8K: Grade School Math Reasoning

GSM8K (Cobbe et al., 2021) is 8,500 grade school math problems requiring 2–8 step reasoning chains. Each problem is a word problem solved through arithmetic operations.

Example problem:

Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did she sell altogether in April and May?

Despite being "grade school" level, GSM8K was hard for early models because it requires tracking state across multiple steps. GPT-3 (2020) scored 17%. GPT-4 scored 92%. Chain-of-thought prompting dramatically improved all models on GSM8K.

MATH benchmark is significantly harder - competition-level mathematics (AMC, AIME problems). GPT-4 scores ~50% with chain-of-thought.

BigBench: Beyond GPT-4 Capability

BigBench (Srivastava et al., 2022) contains 204 diverse tasks contributed by researchers across the field. Critically, tasks were selected to be beyond the expected capability of current models at the time of creation.

Notable BigBench tasks:

• Causal judgment: understanding causal reasoning
• Dyck languages: bracket matching (formal language theory)
• Hyperbaton: unusual word ordering understanding
• Navigate: spatial navigation from instructions
• Social IQ: social reasoning

BigBench Hard (BBH) is a subset of 23 tasks where standard prompting performs near-chance. Chain-of-thought prompting substantially improves performance on BBH, which was used as evidence for the "emergent abilities" hypothesis.

The Benchmark Contamination Problem

How Contamination Happens

1. Public benchmarks are hosted on GitHub, HuggingFace, and in academic papers - all indexed and crawled
2. Large training corpora (Common Crawl, The Pile, RedPajama) include this data
3. Training on a benchmark question is equivalent to giving the model the answer in advance
4. Labs cannot easily audit whether specific benchmark questions appeared in training data for trillion-token corpora

Measuring Contamination

import hashlib
from typing import List, Tuple
import numpy as np

def check_benchmark_contamination(
    training_texts: List[str],
    benchmark_questions: List[str],
    ngram_size: int = 8,
    threshold: float = 0.3,
) -> dict:
    """
    Check if benchmark questions have significant n-gram overlap
    with training data. High overlap suggests contamination.

    Args:
        training_texts: Sample of training data to check against
        benchmark_questions: Benchmark test questions
        ngram_size: Size of n-grams to compare
        threshold: Fraction of benchmark n-grams that must match to flag

    Returns:
        Contamination report
    """
    def get_ngrams(text: str, n: int) -> set:
        tokens = text.lower().split()
        return set(tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1))

    # Build n-gram index from training data
    print("Building training n-gram index...")
    training_ngrams = set()
    for text in training_texts:
        training_ngrams.update(get_ngrams(text, ngram_size))

    print(f"Training n-gram index size: {len(training_ngrams):,}")

    # Check each benchmark question
    contaminated = []
    contamination_ratios = []

    for i, question in enumerate(benchmark_questions):
        question_ngrams = get_ngrams(question, ngram_size)

        if not question_ngrams:
            continue

        overlap = question_ngrams.intersection(training_ngrams)
        ratio = len(overlap) / len(question_ngrams)
        contamination_ratios.append(ratio)

        if ratio > threshold:
            contaminated.append({
                "question_idx": i,
                "question_preview": question[:100],
                "contamination_ratio": round(ratio, 4),
                "matched_ngrams": len(overlap),
                "total_ngrams": len(question_ngrams),
            })

    return {
        "total_questions": len(benchmark_questions),
        "potentially_contaminated": len(contaminated),
        "contamination_rate": round(len(contaminated) / len(benchmark_questions), 4),
        "mean_contamination_ratio": round(np.mean(contamination_ratios), 4),
        "contaminated_examples": contaminated[:5],  # Show first 5
    }


def detect_memorization_with_canary(
    model,
    canary_prefix: str,
    canary_suffix: str,
    top_k: int = 50,
) -> bool:
    """
    Test if a model has memorized a specific text.
    The canary was inserted into training data with a unique string.
    If the model can complete the prefix with the correct suffix,
    it has memorized the training data.

    Used by Google and Apple for differential privacy audits.
    """
    # Generate completions
    completions = []
    for _ in range(top_k):
        # In practice: call your model's generate endpoint
        # completion = model.generate(canary_prefix, temperature=1.0)
        # completions.append(completion)
        pass

    # Check if any completion matches the canary suffix
    for completion in completions:
        if canary_suffix in completion:
            return True  # Memorization detected

    return False


def estimate_contamination_score_inflation(
    clean_score: float,
    contamination_rate: float,
    benchmark_accuracy: float = 0.25,  # Random guess for 4-choice MC
) -> dict:
    """
    Estimate how much a benchmark score is inflated by contamination.

    If contamination_rate fraction of questions were memorized,
    and memorized questions get perfect score vs random on uncontaminated,
    estimate the actual capability score.

    This is a simplified model - actual inflation depends on
    whether memorized questions are harder or easier than average.
    """
    # Observed score = contamination_rate * 1.0 + (1 - contamination_rate) * true_score
    # Solving for true_score:
    true_score = (clean_score - contamination_rate) / (1 - contamination_rate)

    inflation = clean_score - true_score

    return {
        "observed_score": clean_score,
        "estimated_true_score": round(max(0, true_score), 4),
        "estimated_score_inflation": round(max(0, inflation), 4),
        "contamination_rate": contamination_rate,
    }


# Running a model through HumanEval
def evaluate_humaneval(
    model_fn,  # Function that takes a prompt and returns code
    n_samples: int = 1,
    timeout: float = 10.0,
) -> dict:
    """
    Evaluate a code generation model on HumanEval.
    Computes pass@k for k=1, k=10, k=100 if enough samples.

    pip install human-eval
    """
    from human_eval.data import write_jsonl, read_problems
    from human_eval.evaluation import evaluate_functional_correctness
    import tempfile
    import os

    problems = read_problems()

    # Generate completions
    samples = []
    for task_id, problem in problems.items():
        for _ in range(n_samples):
            prompt = problem["prompt"]
            completion = model_fn(prompt)
            samples.append({
                "task_id": task_id,
                "completion": completion,
            })

    # Write to temp file and evaluate
    with tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False) as f:
        for sample in samples:
            import json
            f.write(json.dumps(sample) + '\n')
        temp_file = f.name

    try:
        results = evaluate_functional_correctness(
            temp_file,
            k=[1, 10, 100],
            n_workers=4,
            timeout=timeout,
        )
    finally:
        os.unlink(temp_file)

    return results


# Simple mock model for demonstration
def simple_codex_mock(prompt: str) -> str:
    """Mock model that returns a simple completion."""
    # In practice, call your actual model here
    return "    pass  # TODO: implement"

How to Design a Benchmark That Cannot Be Gamed

This is the key interview question and a genuine hard problem. Key principles:

1. Keep It Private Until Evaluation

Never publish the test set before evaluation. Models cannot contaminate what they cannot crawl. The downside: reproducibility requires trusting the evaluation institution.

2. Dynamic Benchmarks

Continuously generate new questions so the test set never stabilizes. Chatbot Arena does this naturally (users write new prompts constantly). For academic benchmarks: generate questions programmatically with a seed, regenerate with a new seed after each evaluation cycle.

3. Procedural Generation

Generate questions from a formal grammar or template where the underlying logic is varied but the format is consistent. Math problems can be generated by instantiating variables in known solution templates. Programming problems can be generated by creating new data transformation specifications.

import random

def generate_math_problem(seed: int) -> dict:
    """
    Generate a unique arithmetic word problem.
    Same template, different numbers - resistant to memorization.
    """
    rng = random.Random(seed)

    names = ["Alice", "Bob", "Carol", "Dave", "Eve"]
    items = ["apples", "books", "coins", "stickers", "tokens"]

    name1 = rng.choice(names)
    name2 = rng.choice([n for n in names if n != name1])
    item = rng.choice(items)
    initial = rng.randint(10, 100)
    multiplier = rng.randint(2, 5)
    extra = rng.randint(1, 20)

    question = (
        f"{name1} had {initial} {item}. "
        f"{name2} gave {name1} {multiplier} times as many {item}. "
        f"Then {name1} gave away {extra} {item}. "
        f"How many {item} does {name1} have now?"
    )

    answer = initial + multiplier * initial - extra

    return {
        "question": question,
        "answer": answer,
        "seed": seed,
    }

# Generate 1000 unique problems
problems = [generate_math_problem(seed) for seed in range(1000)]
# New benchmark generation: same code, different seed range
new_problems = [generate_math_problem(seed) for seed in range(1000, 2000)]

4. Capability Probes, Not Fact Recall

Design questions that require reasoning, not recall. A model that has seen a question before can still fail a reasoning question if it does not understand the underlying principle. Fact-retrieval questions are much easier to game through memorization.

5. Out-of-Distribution Assessment

Include examples that are deliberately out of distribution from expected training data - novel scenarios, unusual combinations, recently created content. These are harder to contaminate and harder to overfit to.

Production Engineering Notes

Using lm-evaluation-harness

The standard tool for running benchmarks:

# pip install lm-eval
# Run from command line:
# lm_eval --model hf --model_args pretrained=gpt2 --tasks mmlu --num_fewshot 5 --output_path results/

# Or programmatically:
import lm_eval
from lm_eval.models.huggingface import HFLM
from lm_eval.evaluator import evaluate
from lm_eval.tasks import TaskManager

def run_benchmark_evaluation(
    model_name: str,
    tasks: list = ["mmlu", "hellaswag", "arc_challenge"],
    num_fewshot: int = 5,
) -> dict:
    """
    Run standard benchmark evaluation using lm-eval-harness.
    """
    model = HFLM(pretrained=model_name)

    task_manager = TaskManager()
    results = evaluate(
        lm=model,
        task_manager=task_manager,
        tasks=tasks,
        num_fewshot=num_fewshot,
        batch_size="auto",
    )

    # Print summary
    for task, metrics in results["results"].items():
        print(f"{task}: {metrics}")

    return results

Interpreting Benchmark Numbers Responsibly

When reporting benchmark numbers:

• Always specify the exact evaluation setup (few-shot count, temperature, sampling strategy)
• Report confidence intervals when possible (especially for small benchmarks)
• Note if the benchmark test set was accessible during training
• Report multiple benchmarks, not cherry-picked ones
• Include standard deviation across multiple runs for stochastic models

:::danger Never Report a Single Benchmark as "Capability" MMLU measures knowledge breadth. HumanEval measures code generation. HELM measures multiple things simultaneously. No single benchmark measures overall LLM capability. A model that scores highest on MMLU may perform badly on HumanEval and vice versa. Always report a portfolio of benchmarks. :::

:::warning Benchmark Saturation Is Real If the state of the art on a benchmark exceeds 85-90% accuracy, that benchmark is saturating. Differences at the top (88% vs 89%) are often within noise, may be explained by contamination differences, and may not correspond to real-world capability differences. When a benchmark saturates, it should be retired in favor of harder alternatives. :::

Common Mistakes

:::danger Cherry-Picking Benchmarks Reporting only the benchmarks where your model excels while omitting ones where it underperforms is scientific misconduct. Responsible evaluation reports performance across a comprehensive, pre-specified set of benchmarks. :::

:::warning Not Verifying Evaluation Methodology Different papers compute MMLU differently: different few-shot examples, different answer parsing (taking the first token logprob vs free-form generation), different normalization. Before comparing your model's score to a published baseline, verify you are using the same evaluation code. :::

:::danger Treating Human-Level Performance Claims Literally "Human expert performance" on benchmarks like MMLU is a single data point from a small sample of test-takers under specific conditions. Human performance varies enormously. A model that matches "human expert average" on MMLU may still fail on questions that any practicing professional would answer correctly in context. :::

Interview Q&A

Q1: What does MMLU actually measure and why is it saturating?

MMLU measures breadth of declarative knowledge across 57 academic subjects through multiple-choice questions. High MMLU scores mean the model has absorbed a large amount of academic knowledge from its training data. It is saturating because: (1) frontier models have trained on enough data to have essentially memorized most of the factual content that MMLU tests; (2) benchmark contamination (test questions appearing in training data) inflates scores further; (3) the multiple-choice format is one that modern models have seen extensively and are well-calibrated for. Saturation means the benchmark can no longer distinguish between top models. The response has been to develop harder benchmarks (MMLU-Pro, GPQA, ARC-AGI) and private evaluation sets.

Q2: Explain the pass@k metric for HumanEval.

pass@k measures the probability that at least one of k sampled code completions passes all unit tests for a given problem. If you generate k completions for each problem, it is the probability that at least one is correct. In practice, to get an unbiased estimate: generate n=200 samples, count how many pass (c), then use the formula $1 - \binom{n-c}{k}/\binom{n}{k}$ to estimate pass@k for various values of k. pass@1 reflects a single-shot success rate. pass@10 is useful for agentic systems that can try multiple times. The metric is important because model outputs are stochastic - a model that is right 30% of the time will almost certainly find a correct solution if given 10 tries.

Q3: Why is SWE-Bench a more realistic coding benchmark than HumanEval?

HumanEval uses 164 clean, isolated function-writing problems with clear docstrings and unit tests. Real software engineering involves: understanding large, messy codebases; interpreting vague issue descriptions; making changes that don't break existing tests; navigating dependencies. SWE-Bench uses 2,294 real GitHub issues from popular Python repositories. Success is measured by whether the existing test suite passes after the model's proposed fix. The dramatic score difference - GPT-4 at ~85% pass@1 on HumanEval vs ~4% on SWE-Bench (without scaffolding) - reveals that HumanEval capability does not generalize to real software engineering. SWE-Bench is harder to contaminate, harder to overfit to, and more predictive of real engineering performance.

Q4: What is HELM and why is multi-metric evaluation valuable?

HELM is Stanford's Holistic Evaluation of Language Models - a framework that evaluates models across 42 scenarios using 7 metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Multi-metric evaluation is valuable because single metrics hide important trade-offs. A model might achieve 90% accuracy but be poorly calibrated (overconfident in wrong answers), or it might have good overall accuracy but show 15% lower performance for female versus male entities (fairness issue). HELM surfaces all of these simultaneously, giving a complete capability profile. The trade-off: HELM is expensive to run comprehensively, and aggregating many metrics into a single ranking requires subjective weighting choices.

Q5: How would you design a benchmark for evaluating LLM customer service capabilities that cannot be gamed?

Several key principles: (1) Keep the test set private - never publish the exact questions publicly; use an external auditor to run evaluations. (2) Use real customer transcripts - anonymized actual support conversations have the distribution properties of your production system and are harder to contaminate. (3) Design reasoning-heavy scenarios - include cases that require multi-step reasoning, policy interpretation, and escalation judgment; these cannot be answered correctly through memorization of specific examples. (4) Dynamic refresh - quarterly, replace 25% of evaluation cases with new ones; model developers cannot target a moving target. (5) Multi-dimensional scoring - accuracy alone isn't enough; measure resolution rate, customer satisfaction, escalation correctness, and policy compliance separately. (6) Red-team cases - include adversarial cases designed to elicit policy violations; a model that passes only on normal cases is not safe for deployment.

Q6: A model claims 92% on MMLU but only 60% in your internal domain evaluation. How do you interpret this?

The discrepancy likely reflects one or more of: (1) Domain mismatch - MMLU covers general academic knowledge but your domain may have specialized vocabulary, concepts, or reasoning patterns underrepresented in MMLU; (2) Benchmark contamination - the MMLU score is inflated because test questions appeared in training; (3) Overfitting to MMLU-style questions - the model was specifically fine-tuned on MMLU-like multiple-choice questions and that tuning does not transfer; (4) Format mismatch - your internal evaluation may use open-ended questions rather than multiple-choice, and the model may struggle more with generation than selection. The practical response: trust your internal evaluation for your use case. MMLU is a general-knowledge test; your domain evaluation measures what you actually need. Build a larger, more systematic internal benchmark and use MMLU only as a rough sanity check.

:::tip 🎮 Interactive Playground

Visualize this concept: Try the LLM Benchmark Explorer demo on the EngineersOfAI Playground - no code required.

:::