EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Transformer Self-Attention
Self-Attention Heatmap - The cat sat on the mat
The
cat
sat
on
the
mat
← Key (being attended to) →
Query (what is attending) →
0.25
0.17
0.15
0.14
0.14
0.14
0.15
0.23
0.17
0.15
0.15
0.15
0.15
0.19
0.21
0.16
0.15
0.15
0.15
0.15
0.15
0.21
0.17
0.16
0.21
0.15
0.15
0.15
0.17
0.15
0.15
0.17
0.16
0.17
0.16
0.20
The
cat
sat
on
the
mat
Hover a cell to see attention detail. Click a row to inspect it.
Low → High attention
diagonal = self-attention
Attention Heatmap
Example
PeakedUniform
Row sums (should = 1.0)
The1.000
cat1.000
sat1.000
on1.000
the1.000
mat1.000
Each row shows which tokens a given token "pays attention to" when computing its contextual representation. Rows sum to 1.0 (softmax). Lower temperature = sharper focus.

Transformer Self-Attention - Interactive Visualization

Self-attention is the mechanism that made transformers dominate NLP. Every token simultaneously looks at every other token, and the attention weights determine how much influence each token has on the current token's representation. This heatmap shows exactly those weights - each row sums to 1.0 (softmax), showing where each token looks.

  • Each row = one query token; columns = key tokens it attends to; darker = more attention
  • Rows sum to 1.0 - attention weights are a probability distribution via softmax
  • Adjust temperature: low temp sharpens focus (peaked), high temp spreads attention uniformly
  • See ambiguous sentences like "the bank was flooded" - bank attends to flooded, revealing meaning
  • Click any row to see the full attention distribution as a bar chart
  • Foundation for understanding GPT, Claude, BERT, and every modern language model

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.