EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Attention Complexity & Long Context
Full Attention pattern (n=2048)
key tokens →query
Colored = can attend (scaled display for n=2048)
1MB10MB102MB1GB10GB5122k8k32k128kSequence length →Memory (log)Full O(n²)Sliding O(n·w)Sparse O(n·log n)
Seq length
2k
Full attn mem
16.0 MB
Sliding mem
4.0 MB
80GB limit
143k
Controls
Seq length2k
512128k
Complexity
Full: O(n²) memory
Sliding: O(n·w) memory
Sparse: O(n·log n)
Flash: O(n²) FLOPs, O(n) HBM
Flash Attention doesn't reduce FLOPs - it tiles the computation to avoid materialising the full n×n matrix in HBM. Result: same math, ~3× faster, fits longer contexts on same GPU.

Attention Complexity & Long Context - Interactive Visualization

Standard self-attention scales as O(n²) in both memory and compute - processing 32k tokens requires 4GB just for the attention matrix. This demo visualizes the quadratic bottleneck and compares it to efficient alternatives: sliding window attention (linear), sparse attention, and FlashAttention (IO-aware).

  • Quadratic growth curve - slide the sequence length and watch memory and FLOPs explode past 8k tokens
  • Sliding window attention - see how restricting each token to attend only to local neighbors recovers linear scaling
  • Sparse attention patterns - visualize which token pairs attend to each other in strided and local sparse patterns
  • FlashAttention - understand why fusing the attention kernel and using tiled SRAM access cuts memory 10× without approximation
  • Compare memory requirements of full attention vs each efficient variant at 4k, 32k, and 128k tokens

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.