EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Multi-Head Attention Patterns
Sentence: "The cat sat on the mat" - Layer 1
The
cat
sat
on
the
mat
The
cat
sat
on
the
mat
Head 1 · Syntactic
Click to expand
The
cat
sat
on
the
mat
The
cat
sat
on
the
mat
Head 2 · Positional
Click to expand
The
cat
sat
on
the
mat
The
cat
sat
on
the
mat
Head 3 · Coreference
Click to expand
The
cat
sat
on
the
mat
The
cat
sat
on
the
mat
Head 4 · Long-range
Click to expand
The
cat
sat
on
the
mat
The
cat
sat
on
the
mat
Head 5 · Semantic
Click to expand
The
cat
sat
on
the
mat
The
cat
sat
on
the
mat
Head 6 · Uniform
Click to expand
Controls
SENTENCE
"The cat sat on the mat"
"She saw him leave early"
"Dogs chase cats sometimes"
LAYER
SHOW HEADS
Different heads in the same layer specialise in different relationships. Some learn syntax, others learn coreference or positional patterns. This specialisation emerges from training - it is not programmed in.

Multi-Head Attention Patterns - Interactive Visualization

Multi-head attention runs self-attention multiple times in parallel, with each head free to learn different patterns. In practice, heads specialize: some learn syntactic dependencies, some track coreference, some focus on adjacent tokens, some attend long-range. This visualization shows all 6 heads simultaneously so you can see how they divide linguistic work.

  • Syntactic head: strong attention along subject-verb and noun-adjective dependencies
  • Positional head: Gaussian-shaped band attending to nearby tokens
  • Coreference head: repeated words (like "the") cross-attending to each other
  • Long-range head: everything attends strongly to the sentence-start token
  • Semantic head: content words (nouns, verbs) attending to each other across distance
  • Click any head to expand it to full size for detailed inspection

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.