EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content

:::tip 🎮 Interactive Playground Visualize this concept: Try the LLM Observability demo on the EngineersOfAI Playground - no code required. :::

Monitoring ML Serving in Production

The Silent Degradation Nobody Noticed​

At 2:47 PM on a Wednesday, the model serving infrastructure looked perfectly healthy. CPU: 23%. Memory: 41%. Error rate: 0.02%. All green on the dashboard.

Except one metric nobody was watching: p99 inference latency, which had been climbing since 11 AM. At 11 AM it was 82ms. By 2:47 PM it had reached 340ms - and had been above the 150ms SLA for 4 hours. Tens of thousands of users had been experiencing slow responses. The mobile app had started showing error states. Customer support tickets were accumulating.

The cause, discovered post-mortem: a new container deployment had shared a GPU with the inference pod by mistake. GPU memory fragmentation from the co-tenant was causing CUDA memory allocation to periodically stall. Not enough to cause errors. Enough to push p99 from 82ms to 340ms.

The detection gap was four hours. Not because the monitoring was wrong - it was incomplete. Error rate measures failures. Latency histograms measure slowness. GPU memory fragmentation shows up in latency before it shows up in errors. The team was watching error rate. They were not watching p99 latency.

The fix was straightforward: add p99 latency as a primary alert, configure GPU memory utilization as a secondary signal, and implement per-pod GPU isolation. The more important lesson was operational: ML serving failures almost never announce themselves as errors. They whisper in latency tails and GPU memory graphs, hours before they become user-visible.

Why This Exists - The Observability Gap in ML Systems​

Traditional service monitoring watches three signals: availability (is it up?), latency (is it fast?), and saturation (is it overloaded?). The Google SRE book codifies these as the "Four Golden Signals" (plus error rate).

ML serving introduces additional failure modes that the four golden signals cannot capture:

  • Silent accuracy degradation: the model responds fast and without errors but produces wrong predictions due to input distribution shift
  • GPU memory pressure: cumulative allocation without deallocation causes gradual latency increase before OOM errors
  • Batch size collapse: dynamic batching stops filling batches (often due to a client change), GPU utilization drops from 85% to 5%, cost spikes per prediction
  • Model staleness: the cached model on a replica is an older version than production (deployment race condition), causing metric drift

Comprehensive ML serving monitoring must cover: technical metrics (latency, throughput, errors), resource metrics (GPU utilization, VRAM usage), and ML-specific metrics (prediction distribution, data drift, model version consistency).

Historical Context​

Early ML monitoring was primitive by modern standards - if the prediction endpoint returned HTTP 200 and the response looked like a number, monitoring was satisfied.

Prometheus (2012, originally at SoundCloud; CNCF 2016) became the de facto metrics collection standard for cloud-native systems. Its pull-based model and label cardinality made it suitable for ML serving metrics with high dimensionality (per-model, per-variant, per-endpoint).

NVIDIA's DCGM (Data Center GPU Manager) was released in 2018 and provided programmatic access to GPU telemetry: utilization, memory usage, temperature, SM clock speed, PCIe bandwidth, and error counts. The dcgm-exporter Kubernetes DaemonSet made GPU metrics available to Prometheus, enabling GPU-aware dashboards and alerts.

OpenTelemetry (2019, merger of OpenCensus and OpenTracing) standardized distributed tracing across services. For ML serving pipelines - where a single user request may pass through API gateway → feature computation → model inference → postprocessing → response - distributed tracing became essential for attributing latency to specific stages.

Evidently AI (2021), WhyLogs (2021), and Arize AI (2021) emerged as purpose-built ML monitoring platforms addressing the model-specific signals: feature drift, prediction drift, data quality monitoring, and model performance tracking.

The ML Serving Monitoring Stack​

Core Serving Metrics Implementation​

# serving_metrics.py - comprehensive ML serving instrumentation
from prometheus_client import (
    Counter, Histogram, Gauge, Summary,
    start_http_server, REGISTRY
)
import time
import functools
from typing import Callable
import asyncio

# =============================================
# Request-level metrics
# =============================================

inference_requests_total = Counter(
    'ml_inference_requests_total',
    'Total inference requests processed',
    ['model_name', 'model_version', 'endpoint', 'status']
)

inference_latency_seconds = Histogram(
    'ml_inference_latency_seconds',
    'End-to-end inference latency (client-visible)',
    ['model_name', 'model_version', 'endpoint'],
    # Buckets chosen to cover 1ms to 30s range with good resolution
    buckets=[0.001, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250,
             0.500, 1.0, 2.5, 5.0, 10.0, 30.0]
)

model_inference_latency_seconds = Histogram(
    'ml_model_inference_latency_seconds',
    'Pure model forward pass latency (excludes network, preprocessing)',
    ['model_name', 'model_version'],
    buckets=[0.001, 0.002, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500]
)

preprocessing_latency_seconds = Histogram(
    'ml_preprocessing_latency_seconds',
    'Input preprocessing latency',
    ['model_name', 'endpoint'],
    buckets=[0.001, 0.005, 0.010, 0.025, 0.050]
)

# =============================================
# Batch metrics
# =============================================

batch_size_distribution = Histogram(
    'ml_batch_size',
    'Distribution of inference batch sizes',
    ['model_name'],
    buckets=[1, 2, 4, 8, 16, 32, 64, 128]
)

queue_depth = Gauge(
    'ml_inference_queue_depth',
    'Current number of requests waiting in inference queue',
    ['model_name']
)

queue_wait_seconds = Histogram(
    'ml_queue_wait_seconds',
    'Time request spent waiting in queue before inference',
    ['model_name'],
    buckets=[0.001, 0.005, 0.010, 0.025, 0.050, 0.100, 0.250, 0.500]
)

# =============================================
# Model health metrics
# =============================================

model_version_info = Gauge(
    'ml_model_version_info',
    'Currently loaded model version (label-only metric)',
    ['model_name', 'version', 'loaded_at']
)

model_load_duration_seconds = Histogram(
    'ml_model_load_duration_seconds',
    'Time to load model into GPU memory',
    ['model_name', 'version'],
    buckets=[1, 5, 15, 30, 60, 120, 300]
)

# =============================================
# Prediction distribution (for drift detection)
# =============================================

prediction_score_distribution = Histogram(
    'ml_prediction_score',
    'Distribution of prediction scores/probabilities',
    ['model_name', 'class_label'],
    buckets=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
)

top_prediction_class = Counter(
    'ml_top_prediction_class_total',
    'Count of top predicted class per model',
    ['model_name', 'class_id']
)


def track_inference(model_name: str, model_version: str, endpoint: str):
    """Decorator for tracking inference metrics on any serving function."""
    def decorator(fn: Callable):
        @functools.wraps(fn)
        async def async_wrapper(self, request, *args, **kwargs):
            start = time.perf_counter()
            status = "success"
            try:
                result = await fn(self, request, *args, **kwargs)
                return result
            except Exception as e:
                status = "error"
                raise
            finally:
                latency = time.perf_counter() - start
                inference_requests_total.labels(
                    model_name=model_name,
                    model_version=model_version,
                    endpoint=endpoint,
                    status=status
                ).inc()
                inference_latency_seconds.labels(
                    model_name=model_name,
                    model_version=model_version,
                    endpoint=endpoint
                ).observe(latency)
        return async_wrapper
    return decorator

GPU Metrics with DCGM​

NVIDIA DCGM provides GPU-specific telemetry. Deploy dcgm-exporter as a DaemonSet on GPU nodes:

# dcgm-exporter-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-gpu
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.0-3.2.0-ubuntu22.04
          env:
            - name: DCGM_EXPORTER_KUBERNETES
              value: "true"
            - name: DCGM_EXPORTER_COLLECT_INTERVAL
              value: "5000"   # collect every 5 seconds
          securityContext:
            runAsNonRoot: false
            privileged: true
          volumeMounts:
            - name: pod-resources
              mountPath: /var/lib/kubelet/pod-resources
      volumes:
        - name: pod-resources
          hostPath:
            path: /var/lib/kubelet/pod-resources

Key DCGM metrics for ML serving alerting:

# dcgm_alerts.py - critical GPU metrics and their alert thresholds
DCGM_CRITICAL_METRICS = {
    # GPU memory - running out causes OOM errors
    "DCGM_FI_DEV_FB_USED": {
        "description": "GPU framebuffer memory used (MB)",
        "alert_threshold": "85%",  # alert when > 85% of VRAM used
        "query": "DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)"
    },

    # GPU utilization - low = wasted compute, sustained 100% = saturated
    "DCGM_FI_DEV_GPU_UTIL": {
        "description": "GPU compute utilization percentage",
        "alert_on_low": "< 20% during peak hours",  # batching problem
        "alert_on_high": "> 95% sustained",  # capacity issue
    },

    # GPU temperature - high temps throttle performance and indicate cooling issues
    "DCGM_FI_DEV_GPU_TEMP": {
        "description": "GPU temperature in Celsius",
        "alert_threshold": "> 80°C",
    },

    # SM clock speed - throttling means GPU is overheating or power-limited
    "DCGM_FI_DEV_SM_CLOCK": {
        "description": "GPU SM clock frequency (MHz)",
        "alert_on_drop": "< 80% of base clock",  # indicates throttling
    },

    # PCIe bandwidth - high values mean excessive host-device transfers
    "DCGM_FI_DEV_PCIE_TX_THROUGHPUT": {
        "description": "PCIe transmit throughput (KB/s)",
        "alert_threshold": "> 80% of PCIe bandwidth",
    },

    # NVLink bandwidth (for multi-GPU) - inter-GPU communication bottleneck
    "DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL": {
        "description": "NVLink total bandwidth (KB/s)",
        "useful_for": "detecting tensor parallel communication overhead",
    },

    # XID errors - GPU hardware errors
    "DCGM_FI_DEV_XID_ERRORS": {
        "description": "GPU XID error count",
        "alert_threshold": "> 0",  # any GPU error is critical
    }
}

Distributed Tracing with OpenTelemetry​

For ML serving pipelines with multiple stages (preprocessing → feature lookup → model inference → postprocessing), distributed tracing reveals which stage is slow for which request.

# otel_tracing.py - OpenTelemetry instrumentation for ML serving
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.trace import StatusCode
import time

# Initialize tracer
resource = Resource.create({
    "service.name": "image-classifier",
    "service.version": "1.3.0",
    "deployment.environment": "production"
})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("image_classifier")


async def serve_prediction(request) -> dict:
    """
    Full inference pipeline with distributed tracing.
    Each stage is a separate span - Jaeger shows breakdown.
    """
    # Root span: covers the full end-to-end request
    with tracer.start_as_current_span(
        "inference_request",
        attributes={
            "request.id": request.request_id,
            "model.name": "image-classifier",
            "model.version": "1.3.0",
            "batch.size": 1,
        }
    ) as root_span:

        try:
            # Span 1: Image decoding and preprocessing
            with tracer.start_as_current_span("preprocessing") as prep_span:
                prep_start = time.perf_counter()
                image_tensor = decode_and_preprocess(request.image_bytes)
                prep_latency = (time.perf_counter() - prep_start) * 1000
                prep_span.set_attribute("input.size_bytes", len(request.image_bytes))
                prep_span.set_attribute("latency_ms", prep_latency)

            # Span 2: Feature lookup (user context, metadata)
            with tracer.start_as_current_span("feature_lookup") as feat_span:
                feat_start = time.perf_counter()
                features = await get_request_features(request.metadata)
                feat_latency = (time.perf_counter() - feat_start) * 1000
                feat_span.set_attribute("latency_ms", feat_latency)
                feat_span.set_attribute("cache_hit", features.get("cached", False))

            # Span 3: GPU inference (the core operation)
            with tracer.start_as_current_span("model_inference") as infer_span:
                infer_start = time.perf_counter()
                logits = await run_gpu_inference(image_tensor, features)
                infer_latency = (time.perf_counter() - infer_start) * 1000
                infer_span.set_attribute("latency_ms", infer_latency)
                infer_span.set_attribute("batch_size", image_tensor.shape[0])

            # Span 4: Postprocessing (softmax, top-k, class name lookup)
            with tracer.start_as_current_span("postprocessing") as post_span:
                result = postprocess_predictions(logits)
                post_span.set_attribute(
                    "top_class", result["predictions"][0]["class_name"]
                )
                post_span.set_attribute(
                    "confidence", result["predictions"][0]["confidence"]
                )

            root_span.set_attribute("status", "success")
            return result

        except Exception as e:
            root_span.record_exception(e)
            root_span.set_status(StatusCode.ERROR, str(e))
            raise

What Distributed Tracing Reveals​

A Jaeger trace for a slow request might look like:

inference_request [total: 487ms]
  ├── preprocessing [8ms] - normal
  ├── feature_lookup [412ms] ← BOTTLENECK - Redis timeout?
  ├── model_inference [15ms] - normal
  └── postprocessing [2ms] - normal

Without tracing, you see "487ms inference latency" and assume the model is slow. With tracing, you immediately see the feature lookup is the problem - 85% of latency is waiting for Redis, not running the model. The debugging time drops from hours to minutes.

Drift Detection in Serving​

Model drift happens when production inputs diverge from the training distribution. The model returns fast, low-error results that are nonetheless wrong for the current input distribution.

# drift_detector.py - statistical drift detection in serving
import numpy as np
from scipy import stats
from collections import deque
from typing import Dict, List
import time

class ServingDriftDetector:
    """
    Detects distribution shift in model inputs and predictions
    using reference windows and statistical tests.
    """

    def __init__(
        self,
        reference_stats: Dict,    # stats computed from training/validation data
        window_size: int = 1000,  # compare last N predictions to reference
        drift_pvalue_threshold: float = 0.05  # p < 0.05 → significant drift
    ):
        self.reference = reference_stats
        self.window_size = window_size
        self.p_threshold = drift_pvalue_threshold

        # Sliding window of recent predictions
        self.recent_predictions: deque = deque(maxlen=window_size)
        self.recent_input_means: deque = deque(maxlen=window_size)
        self.recent_confidences: deque = deque(maxlen=window_size)

        self.last_drift_check = time.time()
        self.check_interval = 300  # check every 5 minutes

    def record_prediction(
        self,
        input_mean: float,      # mean pixel value or input feature mean
        prediction_class: int,  # top predicted class
        confidence: float       # confidence score for top prediction
    ):
        """Record prediction metadata for drift tracking."""
        self.recent_predictions.append(prediction_class)
        self.recent_input_means.append(input_mean)
        self.recent_confidences.append(confidence)

        # Periodic drift check
        if time.time() - self.last_drift_check > self.check_interval:
            self.check_drift()
            self.last_drift_check = time.time()

    def check_drift(self) -> Dict[str, dict]:
        """
        Run statistical drift tests. Returns dict of test results.
        Significant p-value (< threshold) means drift detected.
        """
        if len(self.recent_predictions) < self.window_size // 2:
            return {}  # not enough data

        results = {}

        # Test 1: Prediction class distribution shift (chi-squared test)
        if "class_distribution" in self.reference:
            recent_classes = list(self.recent_predictions)
            ref_dist = self.reference["class_distribution"]

            # Count occurrences per class in recent window
            n_classes = len(ref_dist)
            recent_counts = np.bincount(recent_classes, minlength=n_classes)
            expected_counts = np.array(ref_dist) * len(recent_classes)

            # Chi-squared test: are recent class frequencies consistent with reference?
            chi2, p_value = stats.chisquare(recent_counts, expected_counts)
            results["class_distribution_drift"] = {
                "test": "chi_squared",
                "statistic": float(chi2),
                "p_value": float(p_value),
                "drift_detected": p_value < self.p_threshold,
            }

        # Test 2: Confidence score distribution shift (KS test)
        if "confidence_mean" in self.reference:
            recent_conf = list(self.recent_confidences)
            # KS test: is the confidence distribution different from reference?
            # Generate reference distribution (approximate as normal)
            ref_mean = self.reference["confidence_mean"]
            ref_std = self.reference["confidence_std"]
            ref_sample = np.random.normal(ref_mean, ref_std, len(recent_conf))

            ks_stat, p_value = stats.ks_2samp(recent_conf, ref_sample)
            results["confidence_drift"] = {
                "test": "kolmogorov_smirnov",
                "statistic": float(ks_stat),
                "p_value": float(p_value),
                "recent_mean": float(np.mean(recent_conf)),
                "reference_mean": ref_mean,
                "drift_detected": p_value < self.p_threshold,
            }

        # Test 3: Input feature distribution shift (KS test on input means)
        if "input_mean" in self.reference:
            recent_means = list(self.recent_input_means)
            ref_mean = self.reference["input_mean"]
            ref_std = self.reference["input_std"]
            ref_sample = np.random.normal(ref_mean, ref_std, len(recent_means))

            ks_stat, p_value = stats.ks_2samp(recent_means, ref_sample)
            results["input_drift"] = {
                "test": "kolmogorov_smirnov",
                "statistic": float(ks_stat),
                "p_value": float(p_value),
                "recent_mean": float(np.mean(recent_means)),
                "reference_mean": ref_mean,
                "drift_detected": p_value < self.p_threshold,
            }

        # Log drifted signals
        drifted = [k for k, v in results.items() if v.get("drift_detected")]
        if drifted:
            print(f"DRIFT DETECTED in: {drifted}")
            # Push to alerting system
            for signal in drifted:
                drift_detected_total.labels(
                    model_name="image-classifier",
                    signal=signal
                ).inc()

        return results

Alert Configuration​

# ml-serving-alerts.yaml - Prometheus alerting rules
groups:
  - name: ml_serving_latency
    rules:
      # P99 latency SLA breach - highest priority
      - alert: MLServingP99LatencySLABreach
        expr: |
          histogram_quantile(0.99,
            rate(ml_inference_latency_seconds_bucket[5m])
          ) > 0.150
        for: 2m
        labels:
          severity: critical
          team: ml-platform
        annotations:
          summary: "P99 inference latency above 150ms SLA"
          description: >
            Model {{ $labels.model_name }} p99 latency is
            {{ $value | humanizeDuration }}, exceeding 150ms SLA.

      # Sudden latency spike (3× baseline in 5 minutes)
      - alert: MLServingLatencySpike
        expr: |
          histogram_quantile(0.95,
            rate(ml_inference_latency_seconds_bucket[5m])
          ) > 3 * histogram_quantile(0.95,
            rate(ml_inference_latency_seconds_bucket[30m] offset 30m)
          )
        for: 3m
        labels:
          severity: warning

  - name: ml_gpu_health
    rules:
      # GPU memory near capacity - OOM errors imminent
      - alert: GPUMemoryHighUsage
        expr: |
          DCGM_FI_DEV_FB_USED /
          (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "GPU memory > 90% utilized"

      # Low GPU utilization during business hours - batching issue
      - alert: GPUUtilizationLow
        expr: |
          DCGM_FI_DEV_GPU_UTIL < 20
          and hour() >= 9 and hour() <= 21  # business hours only
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "GPU utilization < 20% during peak hours"
          description: "Possible batching misconfiguration or traffic drop"

      # Any GPU hardware error
      - alert: GPUXIDError
        expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "GPU XID hardware error detected"

  - name: ml_model_health
    rules:
      # Error rate spike
      - alert: MLServingErrorRateHigh
        expr: |
          rate(ml_inference_requests_total{status="error"}[5m]) /
          rate(ml_inference_requests_total[5m]) > 0.02
        for: 5m
        labels:
          severity: critical

      # Inference queue growing (scaling needed)
      - alert: MLInferenceQueueGrowing
        expr: ml_inference_queue_depth > 100
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Inference queue depth > 100"
          description: "Consider scaling up replicas or increasing batch size"

Grafana Dashboard Structure​

A production ML serving dashboard should have four sections:

Section 1: Request Health (top row)

  • Request rate (QPS) - time series
  • Error rate percentage - time series with threshold line
  • P50/P95/P99 latency - overlaid time series

Section 2: Batch Efficiency

  • Batch size distribution - histogram
  • Queue depth - time series
  • GPU utilization - time series (target: 70-85%)

Section 3: GPU Resource Health

  • GPU memory utilization - gauge + time series
  • GPU temperature - time series
  • SM clock frequency (detect throttling) - time series

Section 4: Model Health

  • Active model versions per replica - table
  • Model load duration (for newly started pods) - histogram
  • Prediction score distribution - histogram (compare vs reference)

Common Mistakes​

:::danger Alerting on Average Latency Instead of Percentiles Average latency hides tail behavior. If 95% of requests take 10ms and 5% take 2 seconds, average is 109ms - looks fine. P99 is 2 seconds - critical SLA breach. Always alert on p95 and p99 latency, not mean. The mean is useless for SLA-governed services. :::

:::danger Not Separating Model Latency from Pipeline Latency Total request latency includes network, preprocessing, feature lookup, model inference, and postprocessing. If you only measure total latency, you cannot tell whether a spike comes from the model or from feature lookup. Use distributed tracing with per-stage spans, or at minimum separate model_inference_latency from total_request_latency as distinct metrics. Engineers waste hours debugging "slow model inference" that turns out to be slow feature retrieval. :::

:::warning Ignoring GPU Utilization as a Batching Health Signal Low GPU utilization (less than 30%) during peak hours is almost always a batching problem - requests are processed one at a time or in tiny batches. This does not cause errors, but it causes each prediction to cost 5-10× what it should. Most teams only alert on GPU utilization being too high (capacity constraint), not too low (batching inefficiency). Both matter for cost and SLA. :::

:::warning Not Tracking Model Version Consistency Across Replicas In a rolling deployment, different pods may run different model versions simultaneously. If some replicas are on v1.2 and others on v1.3 and they have different prediction behaviors, your per-model metrics will appear noisy and your A/B tests will be contaminated. Track the deployed model version as a label on every metric. Alert if more than two model versions are active simultaneously (deployment stuck in partial rollout). :::

Interview Q&A​

Q1: What are the most important metrics to monitor for an ML inference serving system?

A: I divide them into three tiers. Tier 1 (alert immediately): p99 latency against SLA (reveals tail problems before users complain), error rate (obvious failures), and GPU memory utilization above 85% (OOM errors incoming). Tier 2 (alert in 5-15 minutes): p95 latency trend (sustained increase means something is wrong before p99 breaches), inference queue depth (growing queue means insufficient capacity), GPU utilization below 20% during peak (batching problem wasting money). Tier 3 (weekly review): prediction score distribution vs reference (drift detection), class distribution shifts, model version consistency across replicas. The most commonly missed critical metric is p99 latency - teams watch error rate and average latency, miss the fact that 1% of users are getting 10× slower responses than normal.

Q2: How do you detect model serving degradation using distributed tracing?

A: Instrument each stage of the serving pipeline as a separate OpenTelemetry span: preprocessing, feature lookup, model inference, postprocessing. Send spans to Jaeger or Tempo. When p99 latency spikes, the trace for a slow request immediately shows which stage is slow - in 90% of cases, it is not the model inference itself. Common findings: feature lookup latency spikes (Redis slow queries, cache miss storm after a deployment), preprocessing spikes (new input format causing expensive error handling), postprocessing bottleneck (class name lookup hitting a slow database). Without tracing, you see "inference is slow" and start profiling the model. With tracing, you see "feature lookup is 400ms for 15% of requests" and immediately know to check Redis slow log.

Q3: What is the difference between model drift and data drift, and how do you detect each in serving?

A: Data drift (also called covariate shift or input drift) is when the distribution of model inputs changes - different image subjects, different text topics, different user demographics. Data drift does not require ground truth to detect - you can compare the distribution of input features or embeddings against a reference distribution from training time using KS tests or chi-squared tests on the serving metrics. Model drift (also called concept drift) is when the relationship between inputs and correct outputs changes - the world has changed, and the model's learned patterns are now wrong. Model drift requires ground truth labels to detect - you need to compare recent predictions against actual outcomes. This is harder because ground truth is often delayed (you find out tomorrow whether yesterday's recommendation was good) or expensive to collect. In practice, use data drift as an early warning signal (triggers human review) and delayed model quality metrics as the definitive drift signal.

Q4: How does DCGM help you monitor GPU health, and what metrics matter most for ML serving?

A: DCGM (Data Center GPU Manager) provides programmatic access to NVIDIA GPU telemetry that goes far beyond what nvidia-smi shows. The metrics that matter most for ML serving: (1) DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE - VRAM utilization, the most critical metric (high → OOM errors imminent); (2) DCGM_FI_DEV_GPU_UTIL - compute utilization (target 70-85%; below 20% = batching issue, sustained 100% = capacity issue); (3) DCGM_FI_DEV_GPU_TEMP - temperature (above 80°C → thermal throttling reduces performance); (4) DCGM_FI_DEV_SM_CLOCK - SM clock frequency (drop from boost clock indicates power or thermal throttling); (5) DCGM_FI_DEV_XID_ERRORS - GPU hardware errors (non-zero = investigate immediately, often indicates a faulty GPU). Deploy dcgm-exporter as a Kubernetes DaemonSet on GPU nodes to scrape these into Prometheus automatically.

Q5: How would you set up monitoring to catch the "4-hour silent degradation" scenario from the opening - catching serving degradation before users notice?

A: Three layers working together. First, define a p99 latency SLA and alert on it directly: if p99_latency > SLA_threshold for 2 minutes → page on-call. This catches degradation within 2 minutes of it happening, not after 4 hours. The key mistake in the scenario was only watching error rate. Second, set up a latency anomaly alert: if p99_latency > 3× p99_latency_30_minutes_ago → warning. This catches gradual drift even before the absolute SLA threshold is breached. Third, add GPU memory utilization as a leading indicator: memory fragmentation shows up in DCGM metrics before it causes latency increases. DCGM_FI_DEV_FB_USED / total > 0.85 warrants investigation even if latency is currently fine. In the specific scenario (GPU co-tenancy), the correct prevention is resource isolation (Kubernetes GPU request=limit, never share GPU across pods), but the monitoring changes alone would have caught it within 5 minutes instead of 4 hours.

© 2026 EngineersOfAI. All rights reserved.