EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Latency vs Throughput in ML Serving
Latency vs Throughput
2 replicas - 80ms inference - capacity 25 RPS
Latency vs Throughput (RPS)
knee061319250379775941139215189Throughput (RPS)Latency (ms)p50p95p99
Little's Law: L = λ × W (queue length = arrival rate × wait time)
Utilization ρ = λ / μ - latency spikes above ρ = 0.7
Latency Breakdown (baseline, no queuing)
Model inference80ms (77%)
Feature retrieval15ms (14%)
Network overhead5ms (5%)
Pre/post-process4ms (4%)
Total: 104ms (p50 baseline)
Configuration
Replicas2
Inference time80ms
Options
Live Metrics
Capacity: 25 RPS
Knee: 18 RPS
p50 @ knee: 287ms
p99 @ knee: 1342ms
Design Rule
Keep utilization below 70%. Above 80%, p99 latency explodes non-linearly. The "knee" is your capacity limit in practice.

Latency vs Throughput in ML Serving - Interactive Visualization

Latency and throughput have a non-linear relationship in ML serving systems. As throughput approaches capacity, latency increases exponentially - this is the M/M/1 queue behavior. The 'knee of the curve' is around 70% utilization: above that, p99 latency explodes. Little's Law (L = λW) governs queue depth. Adding replicas shifts the saturation point right but does not change the shape of the curve. Caching reduces effective load on the model serving layer.

  • Little's Law: queue length L = arrival rate λ × wait time W - all three scale together
  • The knee of the curve is at ~70% utilization - design for peak below this threshold
  • p99 latency is 3–5× p50 at moderate load, and 10–50× at high utilization
  • Adding replicas scales capacity linearly - the safest way to reduce p99 without code changes

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.