EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Encoder-Only, Decoder-Only, and Encoder-Decoder Architectures
Decoder-only
GPT-4, LLaMA, Mistral, Claude
Token Embeddings
Masked Self-Attention
Add & LayerNorm
Feed-Forward (FFN)
Add & LayerNorm
Next-Token Distribution
Attention type legend
Bidirectional Self-Attention
Masked Self-Attention
Cross-Attention
Best suited for
Text generationChat / instruction followingCode completionReasoning (CoT)
Controls
Architecture
Highlight attention
Key difference: Encoder sees all tokens (bidirectional). Decoder sees only past tokens (causal mask). Enc-Dec adds cross-attention so the decoder can attend to encoder states.
Cross-attention (enc-dec only): Keys and Values come from the encoder; Query comes from the decoder - this is how the decoder "reads" the source.

Encoder-Only, Decoder-Only, and Encoder-Decoder Architectures - Interactive Visualization

The three transformer architecture families serve different tasks. Encoder-only models (BERT, RoBERTa) use bidirectional self-attention and excel at classification, NER, and sentence similarity. Decoder-only models (GPT-4, LLaMA, Mistral) use causal masked attention for autoregressive text generation. Encoder-decoder models (T5, BART) use cross-attention to let the decoder read encoder states, making them ideal for translation and summarization.

  • Interactive architecture diagrams for all three families with color-coded attention layers
  • Attention type highlighting: bidirectional self-attention, masked self-attention, cross-attention
  • Task suitability badges: which architecture excels at classification vs generation vs seq2seq
  • Cross-attention explainer showing how decoder queries attend to encoder keys and values

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.