EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Inference Batching and Throughput
Throughput
0.0 req/s
P50 Latency
0ms
P99 Latency
0ms
GPU Util
0%
Queue Depth
0
Request Queue0
Press Play to start
→
GPU Batch Processing
Idle
→
Completed
None yet
Tradeoff: Larger batch size = higher GPU utilization + throughput, but requests must wait longer to fill the batch - increasing P99 latency. Watch how increasing batch size from 1 to 16 changes all four metrics simultaneously.
Batching Controls
120
116
How it works:

Larger batches improve GPU utilization and throughput - the GPU does more work per second. But requests must wait for the batch to fill, increasing latency. The sweet spot depends on your SLA.

Key formula:
Batch time = 400ms + 60ms × N requests

Inference Batching and Throughput - Interactive Visualization

Serving LLMs efficiently requires balancing two competing goals: minimize latency (individual request wait time) and maximize throughput (requests processed per second). Larger batches amortize GPU overhead and maximize utilization - but requests must wait for the batch to fill. This real-time simulation shows the tradeoff with live P50/P99 latency and GPU utilization metrics.

  • Batch size 1: minimum latency, poor GPU utilization (~12%), low throughput
  • Batch size 16: high GPU utilization (~95%), high throughput, higher P99 latency
  • See queue depth build when arrival rate exceeds processing capacity
  • P50 vs P99 latency: understand why tail latency matters more than average
  • GPU utilization metric: GPUs are matrix multiply engines - small batches waste them
  • Used in: vLLM continuous batching, TensorRT-LLM, NVIDIA Triton serving framework

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.