Continuous Batching & PagedAttention - Interactive Visualization

Static batching waits for a full batch before starting, wasting GPU time when sequences finish early. Continuous batching inserts new requests the moment a slot frees, dramatically improving throughput. PagedAttention eliminates KV cache fragmentation by storing cache in non-contiguous memory pages, like OS virtual memory. Together they power vLLM.

• Static batching timeline: GPU sits idle when early-finishing sequences create empty slots
• Continuous batching: new requests inserted immediately as slots open, GPU stays busy
• KV cache fragmentation: standard allocation wastes memory on reserved but unused space
• PagedAttention: non-contiguous memory pages eliminate fragmentation and increase batch size

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.