Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Generated Tokens

The

quick

brown

fox

jumps

over

the

lazy

dog

and

then

ran

away

quickly

Token

Key K

Value V

Status

Cache is empty. Click "Next Token" to start generating.

WITHOUT CACHE

ops (recomputes all, O(n²))

WITH CACHE

ops (new token only, O(n))

Controls

MAX CONTEXT8

412

Exceeding limit triggers cache eviction (oldest token dropped).

STATUS

Ready - press Next Token

Without KV cache, each new token must recompute attention over all previous tokens - O(n²) work. The cache stores K and V for each past token, so only the new token's K,V need computing - O(n) per step. This is why KV cache size limits how long a context window can be.

KV Cache Explained - Interactive Visualization

The KV cache is the single most important optimization for LLM inference. Without it, generating each new token requires recomputing attention over all previous tokens - O(n²) work per token. With the cache, only the new token's key and value are computed, while all previous pairs are retrieved from memory - O(n) per step. This visualization shows exactly why the cache fills up and starts evicting older context.

Step through generation: each token adds a new K,V row to the cache
See the operations saved counter tick up: cache saves n² - n ops per step
Memory calculator: approximate cache size in MB grows linearly with tokens
Context length slider: see cache eviction when the limit is reached
Without cache vs with cache: quadratic vs linear growth in compute
Why 128K context costs more than 8K: KV cache memory scales with context length

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.