:::tip 🎮 Interactive Playground Visualize this concept: Try the Latency vs Throughput demo on the EngineersOfAI Playground - no code required. :::

Inference Scaling

The Product Launch That Almost Broke Everything

It is 9:00 AM on a Thursday. The product launches in 30 minutes. The ML infrastructure team has pre-scaled inference to 10 replicas, expecting 2× normal traffic. At 9:30 AM, the CEO tweets about the launch. It goes viral. Traffic spikes to 50× normal within 8 minutes. The inference pods max out their CPU limits. The GPU memory is saturated. The load balancer is returning 503s.

The on-call engineer opens the Kubernetes dashboard. The Horizontal Pod Autoscaler (HPA) based on CPU utilization is scaling - but slowly. Each new pod takes 4 minutes to start (container pull + model download + GPU warm-up). By the time a new pod is ready, the spike has already been going for 4 minutes and thousands of requests have failed.

The lesson learned: standard Kubernetes HPA based on CPU/memory is the wrong tool for ML inference scaling. ML pods have long cold-start times (model loading), the right scaling signal is GPU utilization or inference queue depth (not CPU), and scaling needs to be predictive (pre-scale before the spike, not during it).

The team reworks the setup: KEDA with a custom Prometheus metric (inference queue depth), pre-pull model images on all nodes, model weights cached in host DRAM, GPU warm-up deferred to background, and a scheduled scaling rule for the next launch (pre-scale 30 minutes ahead). The next launch scales from 10 to 200 replicas in 3 minutes, absorbs a 100× traffic spike, and the p99 latency never exceeds 95ms.

This lesson covers how to build that system.

---

Why This Exists - The Unique Challenges of ML Scaling

CPU-based services are relatively easy to autoscale: measure CPU utilization, add pods when it exceeds 70%. CPU is a direct proxy for load. Pods start in seconds (container pull is fast, no warm-up needed).

ML inference has three properties that break this model:

1. GPU is the constraint, not CPU. ML inference pods consume 60-90% of a GPU while using minimal CPU. Standard Kubernetes HPA watching CPU utilization sees low CPU and does not scale - while the actual bottleneck (GPU) is saturated.

2. Pods have long cold-start times. An ML serving pod needs to: pull the container image (~30 seconds if not cached), download model weights from S3 (~60-300 seconds depending on model size), move weights to GPU VRAM (~5-30 seconds), run warm-up inference (~10-30 seconds to prime CUDA JIT caches). Total: 2-6 minutes. Standard autoscaling reacts to current load - by the time the new pod is ready, the spike may be over and requests have already failed.

3. The right scaling signal is not resource utilization. For ML inference, the best scaling signal is inference queue depth or pending requests per second. When the queue is growing, you need more replicas - even if GPU utilization is not yet 100%.

---

Historical Context

Kubernetes Horizontal Pod Autoscaler (HPA) was introduced in Kubernetes 1.2 (2016) and initially only supported CPU and memory metrics. Custom metrics support (via the Custom Metrics API) was added in 1.6 (2017), allowing scaling on application-specific signals like request rate or queue depth.

KEDA (Kubernetes Event-Driven Autoscaling), initiated by Microsoft and Red Hat and donated to CNCF in 2020, brought event-driven scaling to Kubernetes. KEDA can scale on Kafka consumer lag, RabbitMQ queue depth, AWS SQS queue length, Prometheus metrics, and dozens of other sources. For ML inference, Prometheus-based scaling on inference queue depth or pending GPU jobs became the standard pattern.

Knative Serving (2018) solved the cold-start problem for serverless ML by maintaining "warm" (zero-replica) to "active" pools and routing the first request to pre-warmed instances. Its queue-proxy sidecar and request-based scaling metrics influenced how ML serving teams think about reactive scaling.

The "scale to zero" pattern for ML - reducing to zero GPU replicas when idle to save cost - became viable with model weight caching (DRAM or local SSD) and fast model loading (seconds rather than minutes). AWS SageMaker Serverless Inference (2021) and GCP Vertex AI Prediction (2021) both support scale-to-zero for lower-traffic models.

---

Scaling Dimensions

ML inference scaling operates on multiple dimensions simultaneously:

When to Scale Vertically vs Horizontally

Scale vertically when:

• Model does not fit on current GPU VRAM (7B model on 16GB GPU)
• Latency is the constraint (larger GPU completes each request faster)
• Model is difficult to shard horizontally (specific architecture constraints)

Scale horizontally when:

• Throughput is the constraint (latency is fine but QPS is too high for one GPU)
• Model fits on one GPU with headroom
• Geographic distribution improves user latency
• Cost efficiency matters (smaller GPUs are more cost-per-FLOP efficient for many workloads)

For most serving scenarios, horizontal scaling is the right answer once model fits on a GPU.

---

KEDA for ML Inference Scaling

KEDA (Kubernetes Event-Driven Autoscaler) scales deployments based on external metrics. For ML inference, the best metrics are:

1. Inference queue depth: how many requests are waiting for GPU capacity
2. GPU utilization: how loaded the current GPUs are
3. Request rate: incoming QPS vs capacity

Setting Up KEDA with Prometheus

# keda-scaledobject.yaml - scale inference deployment on queue depth
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: image-classifier-scaler
  namespace: ml-serving
spec:
  scaleTargetRef:
    name: image-classifier         # Kubernetes Deployment to scale
  minReplicaCount: 2               # never scale below 2 (SLA requirement)
  maxReplicaCount: 50              # hard cap to control cost
  cooldownPeriod: 120              # wait 2 min before scaling down
  pollingInterval: 15              # check metrics every 15 seconds

  triggers:
    # Primary: inference queue depth
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: inference_queue_depth
        # Scale so that each replica handles at most 10 queued requests
        threshold: "10"
        query: |
          sum(inference_queue_depth{service="image-classifier"})
        authModes: "bearer"

    # Secondary: GPU utilization
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: gpu_utilization
        # Also scale if GPU utilization exceeds 80%
        threshold: "80"
        query: |
          avg(DCGM_FI_DEV_GPU_UTIL{service="image-classifier"})

# inference_server.py - expose queue depth metric for KEDA
from prometheus_client import Gauge, Counter, Histogram, start_http_server
import asyncio

# Metrics exposed to Prometheus → consumed by KEDA
inference_queue_depth = Gauge(
    'inference_queue_depth',
    'Number of requests waiting for inference',
    ['service']
)
inference_requests_total = Counter(
    'inference_requests_total',
    'Total inference requests',
    ['service', 'status']
)
inference_latency = Histogram(
    'inference_latency_seconds',
    'Inference latency',
    ['service'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

class MetricsAwareInferenceServer:
    def __init__(self, service_name: str, model, max_queue_size: int = 1000):
        self.service = service_name
        self.model = model
        self.queue: asyncio.Queue = asyncio.Queue(maxsize=max_queue_size)
        # Start Prometheus HTTP server on port 9090
        start_http_server(9090)

    async def handle_request(self, request) -> dict:
        """Add request to queue and wait for result."""
        future = asyncio.get_event_loop().create_future()
        await self.queue.put((request, future))
        # Update queue depth metric immediately
        inference_queue_depth.labels(service=self.service).set(self.queue.qsize())
        return await future

    async def batch_worker(self):
        """Continuously process requests from queue."""
        while True:
            batch = []
            futures = []

            # Collect up to 32 requests or wait 10ms
            try:
                request, future = await asyncio.wait_for(
                    self.queue.get(), timeout=0.010
                )
                batch.append(request)
                futures.append(future)
            except asyncio.TimeoutError:
                if not batch:
                    continue

            # Drain up to batch_size without waiting
            while len(batch) < 32:
                try:
                    request, future = self.queue.get_nowait()
                    batch.append(request)
                    futures.append(future)
                except asyncio.QueueEmpty:
                    break

            # Update queue depth after draining
            inference_queue_depth.labels(service=self.service).set(self.queue.qsize())

            # Run batch inference
            import time
            start = time.perf_counter()
            try:
                results = self.model.predict_batch([r.data for r in batch])
                latency_s = time.perf_counter() - start
                inference_latency.labels(service=self.service).observe(latency_s)

                for future, result in zip(futures, results):
                    if not future.done():
                        future.set_result(result)
                        inference_requests_total.labels(
                            service=self.service, status="success"
                        ).inc()
            except Exception as e:
                for future in futures:
                    if not future.done():
                        future.set_exception(e)
                        inference_requests_total.labels(
                            service=self.service, status="error"
                        ).inc()

---

Solving Cold Start: Fast Model Loading

The biggest barrier to fast autoscaling is cold start time. A typical cold start sequence:

1. Image pull: 30-120 seconds (container image with CUDA, Python, model server)
2. Model weight download: 60-600 seconds (model from S3 or GCS, depending on size)
3. GPU memory load: 5-60 seconds (copy weights from host DRAM to GPU VRAM)
4. CUDA warm-up: 10-60 seconds (JIT compilation of kernels on first forward pass)
5. Readiness check: 5-10 seconds

Total: 2-15 minutes. Autoscaling cannot react to a spike on a 2-minute timescale.

Strategies to reduce cold start:

# fast_model_loader.py - optimizing model loading for fast cold starts

import os
import time
import torch
from pathlib import Path

class FastModelLoader:
    """
    Optimizes model loading for minimum cold-start time.
    Uses local disk cache, mmap, and GPU-direct loading.
    """

    def __init__(
        self,
        model_name: str,
        local_cache_dir: str = "/model-cache",  # mounted fast local SSD
        s3_bucket: str = None,
    ):
        self.model_name = model_name
        self.cache_dir = Path(local_cache_dir)
        self.s3_bucket = s3_bucket

    def load(self, device: str = "cuda") -> torch.nn.Module:
        """Load model with cold-start optimization."""
        model_path = self.cache_dir / self.model_name

        # Step 1: Check local SSD cache first (vs S3 download)
        if not model_path.exists():
            print(f"Model not in local cache - downloading from S3...")
            start = time.perf_counter()
            self._download_from_s3(model_path)
            print(f"Download: {(time.perf_counter() - start):.1f}s")
        else:
            print(f"Model found in local cache: {model_path}")

        # Step 2: Load with memory-mapped weights (avoids CPU DRAM allocation)
        start = time.perf_counter()
        model = torch.load(
            model_path / "model.pt",
            map_location='cpu',
            # mmap=True  # PyTorch 2.1+: load weights directly without copying
        )
        print(f"CPU load: {(time.perf_counter() - start):.1f}s")

        # Step 3: Move to GPU
        start = time.perf_counter()
        model = model.to(device)
        model.eval()
        torch.cuda.synchronize()
        print(f"GPU transfer: {(time.perf_counter() - start):.1f}s")

        # Step 4: CUDA warm-up (compile JIT kernels, prime cuDNN)
        start = time.perf_counter()
        self._warmup(model, device)
        print(f"Warm-up: {(time.perf_counter() - start):.1f}s")

        return model

    def _warmup(self, model: torch.nn.Module, device: str, n_iters: int = 20):
        """Run synthetic forward passes to warm up CUDA JIT and cuDNN."""
        dummy_input = torch.randn(8, 3, 224, 224, device=device)
        with torch.no_grad():
            for _ in range(n_iters):
                model(dummy_input)
        torch.cuda.synchronize()

    def _download_from_s3(self, dest_path: Path):
        import boto3
        s3 = boto3.client('s3')
        dest_path.mkdir(parents=True, exist_ok=True)
        # Use parallel multipart download for large files
        s3.download_file(
            self.s3_bucket,
            f"models/{self.model_name}/model.pt",
            str(dest_path / "model.pt"),
            Config=boto3.s3.transfer.TransferConfig(
                multipart_threshold=1024 * 25,  # 25MB chunks
                max_concurrency=10,             # 10 parallel parts
            )
        )

Kubernetes DaemonSet for Model Pre-Pull

Pre-pull model images on all nodes before they are needed:

# model-prepull-daemonset.yaml
# Runs on every node to pre-pull model images and cache weights on local SSD
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: model-prepuller
  namespace: ml-serving
spec:
  selector:
    matchLabels:
      app: model-prepuller
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-a100   # only on GPU nodes
      containers:
        - name: prepuller
          image: model-serving:latest
          command:
            - /bin/sh
            - -c
            - |
              # Pull and cache model weights to local NVMe SSD
              python -c "
              from fast_model_loader import FastModelLoader
              loader = FastModelLoader('image-classifier-v1.3',
                                       '/model-cache', 'my-models-bucket')
              loader._download_from_s3(loader.cache_dir / loader.model_name)
              print('Model cached on node')
              "
          volumeMounts:
            - name: model-cache
              mountPath: /model-cache
      volumes:
        - name: model-cache
          hostPath:
            path: /data/model-cache  # NVMe SSD on each node
            type: DirectoryOrCreate

---

Spot Instances for Inference

Spot/preemptible GPU instances cost 60-80% less than on-demand but can be terminated with 2-minute notice. For inference serving, they require interrupt-tolerant architecture.

# spot_instance_manager.py - handling spot instance termination
import boto3
import threading
import requests
import logging
from typing import Callable

class SpotTerminationHandler:
    """
    Monitors for EC2 spot instance termination notices.
    Drains in-flight requests before termination.
    """

    METADATA_URL = "http://169.254.169.254/latest/meta-data/spot/termination-time"
    CHECK_INTERVAL_SECONDS = 5

    def __init__(
        self,
        drain_callback: Callable,  # called when termination notice received
        max_drain_seconds: int = 90  # must complete before 2-min notice expires
    ):
        self.drain_callback = drain_callback
        self.max_drain_seconds = max_drain_seconds
        self.termination_noticed = False
        self._monitor_thread = threading.Thread(
            target=self._monitor_termination, daemon=True
        )

    def start(self):
        self._monitor_thread.start()
        logging.info("Spot termination monitor started")

    def _monitor_termination(self):
        """Poll EC2 instance metadata for termination notice."""
        while True:
            try:
                response = requests.get(
                    self.METADATA_URL,
                    timeout=1
                )
                if response.status_code == 200:
                    # Termination notice received - 2 minutes to drain
                    termination_time = response.text
                    logging.warning(
                        f"Spot termination notice received: {termination_time}"
                    )
                    self.termination_noticed = True
                    self.drain_callback()
                    return
            except requests.exceptions.RequestException:
                pass  # Metadata endpoint not reachable or no notice

            import time
            time.sleep(self.CHECK_INTERVAL_SECONDS)


class GracefulDrainer:
    """
    Drains in-flight requests before pod termination.
    Used with both spot termination and rolling deployments.
    """

    def __init__(self, request_counter):
        self.request_counter = request_counter
        self.draining = False

    def start_drain(self):
        """Stop accepting new requests and wait for in-flight to complete."""
        import time
        logging.info("Starting graceful drain...")
        self.draining = True

        # Wait for in-flight requests to complete
        deadline = time.time() + 90  # 90 seconds max
        while self.request_counter.value > 0:
            if time.time() > deadline:
                logging.warning(
                    f"Drain deadline exceeded: "
                    f"{self.request_counter.value} requests still in flight"
                )
                break
            time.sleep(0.5)

        logging.info("Drain complete")

Spot Instance Strategy

The recommended architecture for spot-augmented serving:

Kubernetes Spot Node Pool configuration:

# node-pool-spot.yaml - GKE spot node pool for inference burst capacity
apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerNodePool
metadata:
  name: gpu-spot-pool
spec:
  clusterRef:
    name: ml-cluster
  location: us-central1
  autoscaling:
    minNodeCount: 0
    maxNodeCount: 20   # up to 20 spot GPU nodes
  nodeConfig:
    machineType: a2-highgpu-1g   # A100 instance
    spot: true                    # preemptible/spot pricing
    oauthScopes:
      - https://www.googleapis.com/auth/cloud-platform
    taints:
      - key: spot-instance
        value: "true"
        effect: NO_SCHEDULE       # only tolerating pods can run here
    labels:
      node-type: spot-gpu

---

Global Load Balancing and Geo-Distributed Serving

For global products, serving ML from a single region means high latency for distant users. Geo-distributed serving runs replicas in multiple regions and routes users to the nearest healthy region.

# global_router.py - geo-aware ML serving router
import time
from typing import Dict, Optional
from dataclasses import dataclass

@dataclass
class RegionEndpoint:
    region: str
    endpoint_url: str
    latency_ms: float = 0.0    # measured p50 latency
    error_rate: float = 0.0    # rolling error rate
    capacity_pct: float = 100.0  # current utilization

class GlobalMLRouter:
    """
    Routes inference requests to the optimal region.
    Considers user location, region health, and capacity.
    """

    def __init__(self, regions: Dict[str, RegionEndpoint]):
        self.regions = regions
        self.health_check_interval = 30  # seconds
        self._start_health_checks()

    def select_region(
        self,
        user_region: str,   # user's detected region (from IP geolocation)
        fallback_regions: list = None
    ) -> RegionEndpoint:
        """
        Select optimal region using:
        1. User's home region if healthy and has capacity
        2. Nearest healthy region otherwise
        3. Any healthy region as last resort
        """
        # Try user's home region first
        home = self.regions.get(user_region)
        if home and self._is_healthy(home):
            return home

        # Find nearest healthy region (simplified - use real latency matrix)
        priority_order = fallback_regions or [
            "us-east-1", "us-west-2", "eu-west-1", "ap-southeast-1"
        ]

        for region_name in priority_order:
            endpoint = self.regions.get(region_name)
            if endpoint and self._is_healthy(endpoint):
                return endpoint

        # All regions degraded - return least-bad option
        return min(
            self.regions.values(),
            key=lambda r: r.error_rate
        )

    def _is_healthy(self, region: RegionEndpoint) -> bool:
        """Region is healthy if error rate < 5% and has capacity."""
        return (region.error_rate < 0.05 and
                region.capacity_pct < 90.0 and
                region.latency_ms < 2000)

    def _start_health_checks(self):
        """Background thread to check region health."""
        import threading

        def check_loop():
            while True:
                for region_name, endpoint in self.regions.items():
                    try:
                        start = time.perf_counter()
                        response = self._ping_health(endpoint.endpoint_url)
                        latency_ms = (time.perf_counter() - start) * 1000
                        endpoint.latency_ms = latency_ms
                        endpoint.error_rate = response.get("error_rate_1m", 0)
                        endpoint.capacity_pct = response.get("utilization_pct", 0)
                    except Exception as e:
                        # Health check failed - mark region as degraded
                        endpoint.error_rate = 1.0
                time.sleep(self.health_check_interval)

        thread = threading.Thread(target=check_loop, daemon=True)
        thread.start()

    def _ping_health(self, endpoint_url: str) -> dict:
        import requests
        response = requests.get(f"{endpoint_url}/health", timeout=5)
        return response.json()

---

Predictive Scaling

For scheduled traffic patterns (known product launches, daily peaks), predictive scaling pre-warms capacity before the spike arrives.

# predictive_scaler.py - schedule-based pre-scaling
from kubernetes import client, config
import schedule
import time
import threading
from datetime import datetime, timedelta

class PredictiveMLScaler:
    """
    Pre-scales ML inference deployments based on known traffic patterns.
    """

    def __init__(self, namespace: str = "ml-serving"):
        config.load_incluster_config()  # runs inside Kubernetes
        self.apps_v1 = client.AppsV1Api()
        self.namespace = namespace

    def scale_deployment(
        self, deployment_name: str, replicas: int, reason: str
    ):
        body = {"spec": {"replicas": replicas}}
        self.apps_v1.patch_namespaced_deployment_scale(
            deployment_name,
            self.namespace,
            body
        )
        print(f"{datetime.now().isoformat()}: Scaled {deployment_name} "
              f"to {replicas} replicas ({reason})")

    def schedule_launch_scaling(
        self,
        deployment_name: str,
        launch_time: datetime,
        pre_scale_replicas: int = 50,
        post_launch_min_replicas: int = 20,
        normal_replicas: int = 5
    ):
        """
        Schedule scaling around a product launch.
        Pre-scale 30 minutes before to ensure warm capacity.
        """
        pre_scale_time = launch_time - timedelta(minutes=30)
        post_scale_time = launch_time + timedelta(hours=4)  # 4h after launch

        def run_at(target_time: datetime, fn: callable, *args):
            delay = (target_time - datetime.now()).total_seconds()
            if delay > 0:
                timer = threading.Timer(delay, fn, args)
                timer.start()
                print(f"Scheduled {fn.__name__} at {target_time.isoformat()}")

        run_at(
            pre_scale_time,
            self.scale_deployment,
            deployment_name, pre_scale_replicas, "pre-launch scale"
        )
        run_at(
            post_scale_time,
            self.scale_deployment,
            deployment_name, post_launch_min_replicas, "post-launch normalize"
        )

    def schedule_daily_patterns(self, deployment_name: str):
        """Scale based on known daily traffic patterns (business hours peak)."""
        # Scale up before morning peak
        schedule.every().monday_to_friday.at("08:30").do(
            self.scale_deployment, deployment_name, 20, "morning pre-scale"
        )
        # Scale down after evening quiet
        schedule.every().day.at("22:00").do(
            self.scale_deployment, deployment_name, 3, "evening scale-down"
        )
        # Weekend reduction
        schedule.every().saturday.at("00:00").do(
            self.scale_deployment, deployment_name, 2, "weekend minimum"
        )

        def run_schedule():
            while True:
                schedule.run_pending()
                time.sleep(30)

        threading.Thread(target=run_schedule, daemon=True).start()

---

Production Engineering Notes

GPU Request and Limit Configuration in Kubernetes

# deployment.yaml - correct GPU resource specification
apiVersion: apps/v1
kind: Deployment
metadata:
  name: image-classifier
spec:
  replicas: 5
  template:
    spec:
      containers:
        - name: serving
          image: image-classifier:v1.3
          resources:
            requests:
              memory: "8Gi"
              cpu: "2"
              nvidia.com/gpu: "1"     # request 1 GPU
            limits:
              memory: "16Gi"
              cpu: "4"
              nvidia.com/gpu: "1"     # limit = request for GPU (must be equal)
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: "all"
            - name: CUDA_VISIBLE_DEVICES
              value: "0"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

For GPU resources in Kubernetes: always set request equal to limit (GPU is not a burstable resource - if your pod uses more GPU memory than available, it gets OOM-killed). The NVIDIA device plugin enforces exclusive GPU access per pod unless you configure MIG or time-sharing.

Monitoring Scale Events

# scale_monitor.py - track scaling events and their impact
from prometheus_client import Gauge, Counter, Histogram

scale_events = Counter(
    'ml_serving_scale_events_total',
    'Total scaling events',
    ['deployment', 'direction', 'reason']
)
replica_count = Gauge(
    'ml_serving_replica_count',
    'Current replica count',
    ['deployment']
)
cold_start_duration = Histogram(
    'ml_serving_cold_start_seconds',
    'Time for new replica to become ready',
    ['deployment'],
    buckets=[30, 60, 120, 240, 480]
)
requests_dropped_during_scale = Counter(
    'ml_serving_dropped_requests_total',
    'Requests dropped during scale-out before new replicas were ready',
    ['deployment']
)

---

Common Mistakes

:::danger Using CPU-Based HPA for GPU Inference Standard HPA watching CPU utilization is wrong for GPU inference. Your serving pod runs at 5% CPU while the GPU is at 95% utilization - HPA sees no scaling signal and does nothing. You scale on GPU utilization, inference queue depth, or pending requests per second. Use KEDA with Prometheus metrics. This single mistake accounts for 90% of ML serving scaling failures at companies making their first production GPU deployment. :::

:::danger Not Accounting for Cold Start in Autoscaling Targets If your model takes 4 minutes to start, setting minReadySeconds=30 in your Deployment means the HPA counts the pod as "ready" after 30 seconds, but the model is not warm yet. Set minReadySeconds to your actual warm-up time, and configure readiness probes that only pass after the model has completed its warm-up inference. Otherwise, the load balancer routes traffic to a "ready" pod that takes 400% longer on the first requests due to CUDA JIT compilation. :::

:::warning Spot Instances Without Graceful Drain Spot instances can be terminated with 2-minute notice. If your serving pods do not have graceful termination handlers, in-flight requests are dropped. A dropped inference request often means: a 0-score recommendation, a failed transaction, or an error returned to a user. Implement a SIGTERM handler that (1) stops accepting new requests, (2) waits for in-flight requests to complete, (3) signals readiness to shut down. Combined with load balancer drain (removing the pod from rotation before termination), this achieves zero dropped requests during spot reclamation. :::

:::warning Scaling Down Too Aggressively Scaling down quickly after a traffic spike is tempting (save costs) but dangerous. If your cooldown period is too short and traffic spikes again immediately after scale-down, you have zero warm capacity and users experience high latency or errors while new pods cold-start. The minimum cooldown period should be at least your pod cold-start time × 2. For ML pods with 4-minute cold starts, use a 10-minute cooldown at minimum. :::

---

Interview Q&A

Q1: Why is standard Kubernetes HPA insufficient for ML inference autoscaling?

A: HPA based on CPU and memory metrics misses the actual bottleneck in GPU inference. ML inference pods use the GPU for computation - they can be CPU-idle at 5% while the GPU is saturated at 95%. HPA watching CPU never triggers scale-out. The correct metrics for ML inference autoscaling are: (a) inference queue depth - if the queue is growing, you need more capacity immediately; (b) GPU utilization - if GPUs are consistently above 70%, you need more replicas; (c) pending request rate - if requests are arriving faster than they can be served, scale out. KEDA solves this by supporting custom Prometheus metrics as scaling triggers. Additionally, HPA's reaction time assumes pod cold start is seconds; ML pods take 2-6 minutes. KEDA + predictive scaling + model caching is the proper architecture.

Q2: How do you achieve fast cold starts for ML serving pods?

A: Cold start time is the sum of four phases, each optimizable separately. Image pull: use DaemonSet pre-puller to pull container images on every node before they are needed - image is available locally when pod schedules. Model weight download: cache model weights on each node's local NVMe SSD (also via DaemonSet or init containers), so pods load from disk (1-5 seconds) instead of S3 (1-5 minutes). GPU transfer: use memory-mapped model loading in PyTorch 2.1+ (mmap=True) to transfer weights directly from disk to GPU without staging in CPU RAM - cuts GPU transfer time by 30-50%. CUDA warm-up: defer full warm-up to background after the pod passes its readiness probe, but run enough warm-up (3-5 forward passes) to prime cuDNN algorithm selection before serving real traffic. With these optimizations, a 1GB model can be ready in under 60 seconds instead of 5 minutes.

Q3: How do you use spot instances for ML inference while maintaining reliability?

A: Spot instances require three mitigations. (1) Baseline on-demand capacity: never scale to zero on-demand instances. Maintain enough on-demand replicas to handle your traffic at non-peak times (typically 30-50% of peak). Spot instances handle burst capacity only. (2) Graceful termination: EC2 gives 2-minute notice before termination. The termination handler should: remove the instance from the load balancer immediately (stop receiving new requests), drain in-flight requests (wait up to 90 seconds), then exit cleanly. In Kubernetes, this is a combination of SIGTERM handling in the application and terminationGracePeriodSeconds: 120 in the pod spec. (3) Diverse instance types: request multiple spot instance types (A10G, T4, V100) so that a single instance type becoming unavailable does not prevent scaling. AWS Spot Fleet and GKE's spot node pool both support multi-instance-type pools.

Q4: Describe the architecture for geo-distributed ML inference serving.

A: Geo-distributed serving runs model replicas in multiple regions and routes users to the nearest healthy one. The components: (1) Global load balancer (Cloudflare, AWS Route 53 latency-based, GCP Global LB) that routes based on user IP geolocation and region health; (2) Regional inference clusters with independent autoscaling in each region; (3) Shared model registry (S3, GCS with cross-region replication) so all regions use the same model version; (4) Regional health endpoints that report latency, error rate, and capacity utilization; (5) Failover policy - if a region's error rate exceeds 5%, the global LB stops routing new requests to it and distributes them to adjacent regions. The hard problem is model consistency: when you deploy a new model version, you must deploy to all regions atomically or accept a window where different users get different model versions. Blue-green deployment per region, with global traffic cut-over only after all regions are healthy on the new version, is the safest approach.

Q5: How does KEDA's event-driven scaling work, and why is it better than reactive HPA for ML workloads?

A: KEDA (Kubernetes Event-Driven Autoscaler) works by installing a custom autoscaler that monitors external event sources and scales Kubernetes Deployments or Jobs based on those events. For ML inference, you configure a ScaledObject that tells KEDA to scale the inference deployment based on a Prometheus metric (inference queue depth). KEDA polls the metric every 15-30 seconds and adjusts the desired replica count to maintain the target metric value per replica. If queue depth is 200 and target is 10 per replica, KEDA sets desired replicas to 20. HPA does the same thing but is limited to CPU/memory from the Kubernetes metrics API. KEDA is better for ML because: (a) it can use queue depth as the scaling signal, which leads scaling demand rather than lagging it; (b) scale-to-zero is first-class (HPA cannot scale below 1 replica); (c) it supports multiple simultaneous triggers (scale on queue depth OR GPU utilization - whichever is more demanding); (d) it supports external event sources (Kafka lag, SQS depth) for batch inference workers, enabling event-driven batch processing without polling.