Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/vLLM Architecture and PagedAttention

Model Size

14 GB

Avail for KV

24 GB

Traditional Concurrency

9 seq

PagedAttention Concurrency

24 seq

vLLM Throughput

2,576 tok/s

KV Cache Memory - Traditional vs PagedAttention

Traditional KV Cache

9 concurrent seqs

60% memory wasted

Pre-allocates max_seq_len * batch_size contiguous block. Short sequences leave 60% of their allocation empty.

PagedAttention (vLLM)

24 concurrent seqs

4% memory wasted

KV cache stored in fixed-size pages (like OS virtual memory). Non-contiguous - pages allocated on demand, released immediately when sequence finishes.

Page Table Visualization - 4 Sequences sharing 20 pages

FREE

Seq 0

Seq 1

Seq 2

Seq 3

Free page

Pages are 16 tokens each. Sequences use non-contiguous pages - like virtual memory. When Seq 0 finishes, its 6 pages are immediately freed for new requests.

Throughput Comparison - 7B, 2048 token sequences

Naive (no batching)

840 tok/s

TGI (static batching)

1,820 tok/s

vLLM (continuous batching)

2,576 tok/s

GPU Memory (40 GB) - Allocation

14GB

24GB

Model Weights (14.0 GB)

KV Cache (paged) (24.0 GB)

Free (reusable pages) (0.0 GB)

Overhead (2.0 GB)

vLLM Controls

GPU Memory

Model Size

Sequence Length

Continuous BatchingShow Page Table

Continuous Batching: as soon as one sequence finishes, a new one starts immediately - no waiting for entire batch to complete. 2–5× throughput gain over static batching.

PagedAttention: eliminates KV cache fragmentation. Enables 3–4× more concurrent sequences on the same GPU memory.

vLLM Architecture and PagedAttention - Interactive Visualization

vLLM (2023, Kwon et al., UC Berkeley) introduced PagedAttention - a GPU memory management technique inspired by OS virtual memory paging. Traditional LLM serving pre-allocates a contiguous KV cache block of size max_sequence_length per request, wasting 60% of GPU memory on average because most sequences are shorter than the maximum. PagedAttention stores KV cache in fixed-size non-contiguous pages (typically 16 tokens per page), allocated on demand and freed immediately when a sequence completes. This nearly eliminates fragmentation, enabling 3–4× more concurrent sequences on the same GPU. Combined with continuous batching - where finished sequences are immediately replaced by new requests without waiting for the entire batch to complete - vLLM achieves 2–5× higher throughput than naive static batching approaches.

PagedAttention: KV cache in 16-token pages, non-contiguous, OS virtual memory analogy - 4% waste vs 60%
Continuous batching: finished sequences release pages immediately, new requests fill slots within the same forward pass
Memory example: 40 GB A100 with 7B model leaves ~26 GB for KV cache - paged gives 3× more concurrent sequences
vLLM throughput: 2800 tok/s for 7B vs 840 tok/s naive - 3.3× improvement from batching + memory efficiency

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.