EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Seq2Seq with Attention
HUD
Decoder step: -
Seq2Seq + Attention
Encoder-decoder with attention heatmap
Source Length
4 tokens
Target Length
4 tokens
Attention Type
Actions
How It Works
The encoder reads all source tokens and produces hidden states e0…eN.

At each decoder step, attention scores each encoder state:
score(d_t, e_i) = d_t · e_i

Weights = softmax(scores). The context vector is a weighted sum of encoder states.

This lets the decoder "look back" at any part of the input - the heatmap shows which encoder tokens each decoder step attends to.

Seq2Seq with Attention - Interactive Visualization

Seq2seq models encode an entire input sequence into a fixed-length context vector, then decode it into an output sequence. Attention solves the bottleneck problem: instead of compressing everything into one vector, the decoder learns to attend to different encoder positions at each step. The attention weight matrix is a soft alignment between input and output tokens - directly interpretable as 'what input is the model looking at right now'.

  • Watch the encoder process input tokens and build up hidden state representations at each position
  • See the context vector bottleneck: without attention, all information must pass through a single fixed-size vector
  • Observe the attention heatmap: bright cells show which input positions receive focus during each output step
  • Understand additive (Bahdanau) vs multiplicative (Luong) attention - different scoring functions for the same idea
  • See how attention was the critical insight that led directly to the Transformer architecture two years later

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.