EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Autoscaling ML Inference with HPA and KEDA
Replicas
2
RPS
0
CPU util
20%
Latency p95
377ms
Status
Stable
Request Rate (RPS) - solid | Replicas×30 - dashed
CPU Utilization %
HPA Spec (CPU-based)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 1
  maxReplicas: 10
  scaleUpCooldown: 3s
  scaleDownCooldown: 8s
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 60
HPA Controls
Load (RPS)
Traffic30 rps
0200
Scale-Up Cooldown
Delay3 ticks
fastslow
Scale-Down Cooldown
Delay8 ticks
fastslow
Scaling Mode
HPA reacts to CPU/memory after load arrives. KEDA reacts to queue depth or RPS before pods are overloaded - 10-30s faster at scale.

Always set a generous scaleDownCooldown to avoid replica flapping.

Autoscaling ML Inference with HPA and KEDA - Interactive Visualization

Model serving pods must scale with traffic to maintain latency SLOs. Kubernetes HPA (Horizontal Pod Autoscaler) automatically adjusts replica count based on CPU utilization or custom metrics. CPU-based HPA reacts after pods are already overloaded - useful but lagging. KEDA (Kubernetes Event-Driven Autoscaling) can trigger on request queue depth or RPS directly, scaling 10-30 seconds faster during traffic spikes. Scale-up and scale-down cooldowns prevent replica flapping. This demo shows live autoscaling response to traffic changes, configurable cooldowns, and HPA vs KEDA behavior.

  • HPA monitors CPU/memory across pods and adds/removes replicas to hit a target utilization (default 80%)
  • KEDA scales on external metrics like Kafka queue depth, SQS message count, or Prometheus RPS
  • Scale-up cooldown (stabilizationWindowSeconds): wait N seconds before adding more replicas to avoid over-scaling
  • Scale-down cooldown: wait longer (default 5 minutes) before removing replicas to handle bursty traffic
  • CPU-based HPA has inherent lag - load must materialize before CPU rises, then HPA reacts
  • KEDA with request-rate metric scales before pods are stressed - better for latency-sensitive inference

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.