Model Evaluation Gates

The Subgroup That Slipped Through

The new credit risk model had cleared every gate. Overall AUC: 0.874. Better than the 0.861 baseline. Average precision: 0.612 vs baseline 0.601. The performance gate was configured to check two metrics - overall AUC and overall average precision - and the new model beat the baseline on both. It was promoted to production automatically at 3 AM on a Thursday.

By Monday, the compliance team had flagged an anomaly. The false-positive rate (incorrectly denying credit to applicants who would have repaid) for applicants in one geographic region was 3.8x higher than for applicants in other regions. The model was performing significantly worse on this subgroup. The aggregate metrics had hidden it: the region represented 6% of the applicant pool, so even a large regression there barely moved the overall numbers.

The bank spent six weeks on a regulatory response. The fix in the ML pipeline took two hours: add subgroup evaluation to the gate check. If any subgroup's false-positive rate is more than 2x the overall false-positive rate, the gate fails. That rule, had it existed, would have caught the problem before the model ever reached production.

This lesson teaches gate design that would have stopped that model.

:::tip 🎮 Interactive Playground Visualize this concept: Try the Model Validation Gates demo on the EngineersOfAI Playground - no code required. :::

Why This Exists

Early ML CI/CD pipelines borrowed the concept of "quality gates" from software CI: a pass/fail check that must succeed before code progresses. The first ML quality gates were simple absolute thresholds: "AUC must be above 0.85." These caught catastrophically bad models.

The limitations became apparent as teams gained experience:

1. Single-metric gates miss tradeoffs. Optimizing AUC can degrade precision, recall, or fairness metrics. A model that improves AUC by overfitting to the majority class passes a single-metric gate while being worse in practice.

2. Absolute thresholds become stale. As models improve over time, the threshold that was ambitious in 2022 becomes trivially easy to beat in 2025. You are not comparing against the right baseline.

3. Aggregate metrics hide subgroup failures. A subgroup representing 5-10% of the population can fail catastrophically without moving aggregate metrics by more than noise.

Modern gate design addresses all three limitations.

The Four Gate Types

Implementing the Gate System

The gate system is a Python script that reads evaluation JSON and makes decisions. It is called from CI (GitHub Actions, GitLab CI) as a final step before model promotion.

# scripts/check_gate.py
"""
Multi-metric model evaluation gate.

Exit codes:
  0 - all gates passed, model approved for promotion
  1 - one or more gates failed, model rejected
  2 - gate configuration error or missing metrics
"""

import json
import sys
import argparse
from dataclasses import dataclass
from typing import Optional
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger(__name__)


@dataclass
class GateResult:
    name: str
    passed: bool
    metric_name: str
    new_value: float
    baseline_value: Optional[float]
    threshold: float
    message: str


def load_metrics(path: str) -> dict:
    with open(path) as f:
        return json.load(f)


def check_absolute_gate(
    metrics: dict,
    metric_name: str,
    min_value: float,
    display_name: str = None
) -> GateResult:
    """Gate type 1: metric must meet absolute minimum."""
    display_name = display_name or metric_name
    value = metrics.get(metric_name)
    if value is None:
        raise ValueError(f"Metric '{metric_name}' not found in evaluation results.")

    passed = value >= min_value
    return GateResult(
        name=f"absolute:{display_name}",
        passed=passed,
        metric_name=metric_name,
        new_value=value,
        baseline_value=None,
        threshold=min_value,
        message=(
            f"PASS: {display_name}={value:.4f} >= {min_value}" if passed
            else f"FAIL: {display_name}={value:.4f} < minimum {min_value}"
        )
    )


def check_regression_gate(
    new_metrics: dict,
    baseline_metrics: dict,
    metric_name: str,
    max_regression: float,
    display_name: str = None
) -> GateResult:
    """Gate type 2: new model must not regress by more than max_regression vs baseline."""
    display_name = display_name or metric_name
    new_value = new_metrics.get(metric_name)
    baseline_value = baseline_metrics.get(metric_name)

    if new_value is None:
        raise ValueError(f"Metric '{metric_name}' not in new model evaluation results.")
    if baseline_value is None:
        logger.warning(f"Metric '{metric_name}' not in baseline. Skipping regression gate.")
        return GateResult(
            name=f"regression:{display_name}", passed=True,
            metric_name=metric_name, new_value=new_value,
            baseline_value=None, threshold=max_regression,
            message=f"SKIP: No baseline for '{metric_name}'"
        )

    regression = baseline_value - new_value
    passed = regression <= max_regression
    return GateResult(
        name=f"regression:{display_name}",
        passed=passed,
        metric_name=metric_name,
        new_value=new_value,
        baseline_value=baseline_value,
        threshold=max_regression,
        message=(
            f"PASS: {display_name} regression={regression:+.4f} <= tolerance {max_regression}"
            if passed else
            f"FAIL: {display_name} regressed {regression:.4f} "
            f"(new={new_value:.4f}, baseline={baseline_value:.4f}), "
            f"tolerance={max_regression}"
        )
    )


def check_subgroup_gates(
    new_metrics: dict,
    baseline_metrics: dict,
    subgroup_prefix: str = "roc_auc_",
    max_regression: float = 0.03,
    max_relative_disparity: float = 2.0,
) -> list[GateResult]:
    """
    Gate type 3: subgroup analysis.
    - Each subgroup metric must not regress by more than max_regression vs baseline
    - No subgroup's false-positive rate can be more than max_relative_disparity × overall FPR
    """
    results = []

    # Find all subgroup metrics (e.g., roc_auc_young, roc_auc_established)
    subgroup_keys = [k for k in new_metrics if k.startswith(subgroup_prefix)]

    for key in subgroup_keys:
        group_name = key[len(subgroup_prefix):]
        result = check_regression_gate(
            new_metrics=new_metrics,
            baseline_metrics=baseline_metrics,
            metric_name=key,
            max_regression=max_regression,
            display_name=f"subgroup:{group_name}"
        )
        results.append(result)

    # Check relative disparity in false-positive rate if available
    overall_fpr = new_metrics.get("fpr_at_threshold")
    if overall_fpr and overall_fpr > 0:
        for key in [k for k in new_metrics if k.startswith("fpr_at_threshold_")]:
            group_name = key[len("fpr_at_threshold_"):]
            group_fpr = new_metrics[key]
            ratio = group_fpr / overall_fpr
            passed = ratio <= max_relative_disparity
            results.append(GateResult(
                name=f"disparity:fpr:{group_name}",
                passed=passed,
                metric_name=key,
                new_value=group_fpr,
                baseline_value=overall_fpr,
                threshold=max_relative_disparity,
                message=(
                    f"PASS: Subgroup '{group_name}' FPR ratio = {ratio:.2f}x (threshold {max_relative_disparity}x)"
                    if passed else
                    f"FAIL: Subgroup '{group_name}' FPR = {group_fpr:.4f} "
                    f"is {ratio:.2f}x the overall FPR {overall_fpr:.4f}. "
                    f"Maximum allowed: {max_relative_disparity}x"
                )
            ))

    return results


def check_operational_gates(
    new_metrics: dict,
    max_latency_p99_ms: float = 100.0,
    max_memory_gb: float = 2.0,
) -> list[GateResult]:
    """Gate type 4: operational/infrastructure metrics."""
    results = []

    if "latency_p99_ms" in new_metrics:
        latency = new_metrics["latency_p99_ms"]
        passed = latency <= max_latency_p99_ms
        results.append(GateResult(
            name="operational:latency_p99",
            passed=passed,
            metric_name="latency_p99_ms",
            new_value=latency,
            baseline_value=None,
            threshold=max_latency_p99_ms,
            message=(
                f"PASS: P99 latency={latency:.1f}ms <= {max_latency_p99_ms}ms"
                if passed else
                f"FAIL: P99 latency={latency:.1f}ms exceeds {max_latency_p99_ms}ms SLA"
            )
        ))

    if "model_size_gb" in new_metrics:
        size = new_metrics["model_size_gb"]
        passed = size <= max_memory_gb
        results.append(GateResult(
            name="operational:model_size",
            passed=passed,
            metric_name="model_size_gb",
            new_value=size,
            baseline_value=None,
            threshold=max_memory_gb,
            message=(
                f"PASS: Model size={size:.2f}GB <= {max_memory_gb}GB limit"
                if passed else
                f"FAIL: Model size={size:.2f}GB exceeds {max_memory_gb}GB limit"
            )
        ))

    return results


def run_all_gates(
    new_metrics_path: str,
    baseline_metrics_path: str,
    config: dict,
) -> tuple[bool, list[GateResult]]:
    """Run all configured gates and return (all_passed, results_list)."""
    new_metrics = load_metrics(new_metrics_path)
    try:
        baseline_metrics = load_metrics(baseline_metrics_path)
    except FileNotFoundError:
        logger.warning("Baseline metrics not found - skipping regression gates.")
        baseline_metrics = {}

    all_results = []

    # Gate 1: Absolute minimums
    for metric, threshold in config.get("absolute_minimums", {}).items():
        all_results.append(check_absolute_gate(new_metrics, metric, threshold))

    # Gate 2: Regression vs baseline
    for metric, max_reg in config.get("regression_tolerances", {}).items():
        all_results.append(check_regression_gate(new_metrics, baseline_metrics, metric, max_reg))

    # Gate 3: Subgroup analysis
    all_results.extend(check_subgroup_gates(
        new_metrics,
        baseline_metrics,
        subgroup_prefix=config.get("subgroup_metric_prefix", "roc_auc_"),
        max_regression=config.get("subgroup_max_regression", 0.03),
        max_relative_disparity=config.get("max_fpr_disparity_ratio", 2.0),
    ))

    # Gate 4: Operational metrics
    all_results.extend(check_operational_gates(
        new_metrics,
        max_latency_p99_ms=config.get("max_latency_p99_ms", 100.0),
        max_memory_gb=config.get("max_model_size_gb", 2.0),
    ))

    all_passed = all(r.passed for r in all_results)
    return all_passed, all_results


def main():
    parser = argparse.ArgumentParser(description="Multi-metric model evaluation gate")
    parser.add_argument("--new-metrics", required=True)
    parser.add_argument("--baseline-metrics", required=True)
    parser.add_argument("--config", default="config/gates.yaml")
    parser.add_argument("--output", default="gate_results.json")
    args = parser.parse_args()

    # Load gate configuration
    import yaml
    with open(args.config) as f:
        config = yaml.safe_load(f)

    all_passed, results = run_all_gates(
        args.new_metrics, args.baseline_metrics, config
    )

    # Print detailed results
    print("\n" + "=" * 60)
    print("MODEL EVALUATION GATE RESULTS")
    print("=" * 60)
    for r in results:
        status = "PASS" if r.passed else "FAIL"
        print(f"[{status}] {r.message}")
    print("=" * 60)

    passed_count = sum(1 for r in results if r.passed)
    print(f"\nResult: {passed_count}/{len(results)} gates passed")

    if all_passed:
        print("\nOVERALL: GATE PASSED - model approved for promotion")
    else:
        failed = [r for r in results if not r.passed]
        print(f"\nOVERALL: GATE FAILED - {len(failed)} gate(s) did not pass")
        for r in failed:
            print(f"  - {r.name}: {r.message}")

    # Write results for CI artifact
    import json
    with open(args.output, "w") as f:
        json.dump({
            "overall_passed": all_passed,
            "gates_passed": passed_count,
            "gates_total": len(results),
            "results": [
                {
                    "name": r.name,
                    "passed": r.passed,
                    "metric": r.metric_name,
                    "new_value": r.new_value,
                    "baseline_value": r.baseline_value,
                    "threshold": r.threshold,
                    "message": r.message,
                }
                for r in results
            ]
        }, f, indent=2)

    sys.exit(0 if all_passed else 1)


if __name__ == "__main__":
    main()

Gate Configuration File

# config/gates.yaml - gate thresholds for the fraud detection model
# Modify these when model architecture changes or business requirements shift

# Gate 1: Absolute minimums - catastrophic failure catches
absolute_minimums:
  roc_auc: 0.88          # Never ship a model with AUC below this
  average_precision: 0.60  # Recall-precision tradeoff minimum
  f1_at_0_5: 0.55        # F1 at default threshold

# Gate 2: Regression tolerances vs current production baseline
regression_tolerances:
  roc_auc: 0.010          # Max AUC regression allowed (1 point)
  average_precision: 0.015  # Max AP regression allowed (1.5 points)
  recall_at_fpr_0_05: 0.02  # Max recall regression at 5% FPR

# Gate 3: Subgroup analysis
subgroup_metric_prefix: "roc_auc_"
subgroup_max_regression: 0.030   # Subgroups can regress up to 3 points
max_fpr_disparity_ratio: 2.0     # No subgroup FPR > 2x overall FPR

# Gate 4: Operational constraints
max_latency_p99_ms: 80.0         # P99 inference latency limit
max_model_size_gb: 1.5           # Max model artifact size

Statistical Significance in Automated Gates

Small evaluation sets can make the gate noisy. A 0.003 AUC difference on 1000 samples may not be statistically significant. For critical gates, add a significance check:

# scripts/significance_check.py
import numpy as np
from scipy import stats


def bootstrap_auc_confidence_interval(
    y_true: np.ndarray,
    y_score: np.ndarray,
    n_bootstrap: int = 1000,
    confidence: float = 0.95,
) -> tuple[float, float]:
    """
    Compute bootstrap confidence interval for AUC.
    Returns (lower_bound, upper_bound).
    """
    from sklearn.metrics import roc_auc_score
    n = len(y_true)
    auc_samples = []
    rng = np.random.default_rng(seed=42)

    for _ in range(n_bootstrap):
        idx = rng.integers(0, n, size=n)  # bootstrap sample with replacement
        try:
            auc = roc_auc_score(y_true[idx], y_score[idx])
            auc_samples.append(auc)
        except ValueError:
            # Can happen if bootstrap sample has only one class
            pass

    alpha = (1 - confidence) / 2
    lower = np.percentile(auc_samples, alpha * 100)
    upper = np.percentile(auc_samples, (1 - alpha) * 100)
    return float(lower), float(upper)


def is_improvement_significant(
    new_auc: float,
    baseline_auc: float,
    new_ci: tuple[float, float],
    baseline_ci: tuple[float, float],
) -> bool:
    """
    Returns True if new model AUC is statistically significantly better
    than baseline (CIs don't overlap substantially).
    """
    # If new model's lower CI bound exceeds baseline's upper CI bound,
    # the improvement is clearly significant
    return new_ci[0] > baseline_ci[1]


# Usage in gate check:
# new_ci = bootstrap_auc_confidence_interval(y_true, new_scores)
# baseline_ci = bootstrap_auc_confidence_interval(y_true, baseline_scores)
# significant = is_improvement_significant(new_auc, baseline_auc, new_ci, baseline_ci)

Gate Failure Responses

Not all gate failures should have the same response. Design failure severity levels:

# Gate failure response matrix
GATE_RESPONSES = {
    # Critical failures: block immediately, page on-call
    "absolute:roc_auc": {
        "action": "block",
        "severity": "critical",
        "notify": ["#ml-alerts", "oncall-ml"],
        "message": "Model fails minimum AUC threshold - do not promote",
    },
    # Subgroup failures: block + require fairness review
    "disparity:fpr:*": {
        "action": "block_require_review",
        "severity": "high",
        "notify": ["#ml-fairness", "#ml-alerts"],
        "message": "Model shows demographic disparity - requires fairness team review before proceeding",
        "create_ticket": True,
    },
    # Regression failures: block + auto-bisect to find culprit commit
    "regression:roc_auc": {
        "action": "block",
        "severity": "medium",
        "notify": ["#ml-ci"],
        "message": "AUC regression vs production baseline",
        "auto_bisect": True,
    },
    # Operational failures: block with infra team notification
    "operational:latency_p99": {
        "action": "block",
        "severity": "medium",
        "notify": ["#ml-infra"],
        "message": "Inference latency exceeds SLA",
    },
}

Production Notes

Fixed evaluation set: The evaluation set must never be resampled between CI runs. If it changes, metric comparisons across runs become meaningless. Version your evaluation sets (eval_set_v1.parquet, eval_set_v2.parquet) and change versions deliberately, with a changelog explaining why.

Storing baseline metrics: Fetch baseline metrics from the model registry at gate-check time. Never hardcode them in the CI YAML - the baseline must reflect the currently deployed model, not whatever was deployed when you wrote the YAML. Use mlflow.get_registered_model or similar to query production model metrics dynamically.

Gate evolution: Document when and why gates were changed. If you lower a threshold because the model architecture changed, record that decision. Unexplained threshold changes are a red flag in compliance audits.

:::tip Visualize Gate Results as a CI Comment Post a formatted gate results summary as a PR/MR comment (GitHub: actions/github-script, GitLab: gitlab-ci comment API). Engineers should be able to see "which gate failed and why" directly in the code review, not by digging through CI logs. :::

:::warning Do Not Lower Gates Under Pressure The fastest path to a production incident is lowering a gate threshold because a sprint deadline is approaching and the model just barely fails. Treat gates as contracts. If the model does not pass, the model does not ship - investigate why and fix it. "We'll fix it after launch" is how subgroup failures end up in regulatory proceedings. :::

:::danger Champion-Challenger Without Regression Gates Some teams implement champion-challenger (new model must beat the current production model) as their only gate. This is a necessary gate but not sufficient. If the champion model degrades over time (data drift), the challenger only needs to beat a degraded baseline. Always include absolute minimum thresholds alongside regression checks. :::

Interview Q&A

Q: Why is a single-metric AUC gate insufficient for ML promotion decisions?

A single AUC gate catches catastrophically bad models but misses three failure modes: (1) a model that improves overall AUC while degrading recall on the minority class, (2) a model that performs well overall but fails on a specific subgroup (which can be invisible in aggregate metrics if the subgroup is small), and (3) a model that passes the AUC threshold but has P99 latency 5x too high for the production SLA. Multi-metric gates check all relevant dimensions of model quality.

Q: How do you compare a new model against the production baseline in an automated gate?

At gate-check time, query the model registry for the current production model's evaluation metrics (stored when that model was registered). Fetch them into a JSON file. The gate script reads both the new model's metrics and the baseline metrics, then checks that regression on each metric does not exceed the configured tolerance. This is dynamic - as the production model improves over time, the bar automatically rises.

Q: What is a subgroup gate and why is it important?

A subgroup gate checks model performance disaggregated by meaningful subgroups (age, geography, user segment) rather than just the aggregate. It is important because aggregate metrics can hide large regressions in minority subgroups. A subgroup that is 5% of your users is nearly invisible in aggregate AUC, but if the model is systematically failing that group, you have a real problem - ethical, business, and regulatory. Subgroup gates surface these failures automatically.

Q: How do you handle statistical significance in automated evaluation gates?

On small evaluation sets (fewer than 10,000 samples), metric differences can be noise rather than signal. Use bootstrap confidence intervals to determine if the gap between new model and baseline is statistically significant. Only block promotion for regressions that are outside the confidence interval - not for noise. For very small subgroups (under 500 samples), be cautious about automated gating and consider requiring human review instead.

Q: What should happen when a gate fails - just block or also notify?

Gate failures should do both. Block the promotion (exit 1 in CI). Post a detailed report as a CI comment. Send a Slack/email notification to the model owner. For certain failure types (subgroup disparity, critical absolute minimum failure), create an incident ticket automatically. The key is that a gate failure is not a silent failure - the right people know about it immediately, with enough detail to diagnose the cause without digging through CI logs.