EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Layer Normalization and Residual Connections
Pre-LN (modern)
Input x
LayerNorm
Self-Attention
Residual Add
LayerNorm
Feed-Forward
Residual Add
Output
GPT-3, LLaMA - more stable training
Post-LN (original)
Input x
Self-Attention
Feed-Forward
Output
Original BERT/GPT - needs careful LR warmup
Normalization effect - 16 activation dimensions
Pre-norm (var=3)
dim 0dim 15
Post-norm (γ=1, β=0)
dim 0dim 15
Input mean
-0.197
Input variance
6.357
Normed mean
-0.000
Learnable parameters γ (gain) and β (bias)
LN(x) = γ · (x − μ) / √(σ² + ε) + β
γ=1.0 rescales the normalized activations. β=0.0 shifts them. Both are learned per-dimension (d_model separate γ and β values).
Residual Stream
The residual connection (x + sublayer(x)) acts as an information highway. Pre-LN applies norm before the sublayer so the residual path is always un-normalized - this stabilizes gradients at depth.
Controls
Placement
Input variance3
110
γ (gain)1.0
β (bias)0.0
Pre-LN normalizes the input to each sub-layer before it sees attention or FFN. Gradients flow cleanly through the un-normalized residual path. Now standard in GPT-3, LLaMA, Mistral.

Layer Normalization and Residual Connections - Interactive Visualization

Layer normalization stabilizes training by normalizing each token's activation vector to zero mean and unit variance, then rescaling with learnable parameters gamma and beta. Pre-LN (normalizing before each sub-layer) is now standard in GPT-3, LLaMA, and Mistral because it allows stable gradient flow through the un-normalized residual path. Post-LN (original BERT placement) requires careful learning rate warmup.

  • Side-by-side Pre-LN vs Post-LN architecture diagram - click to switch between them
  • Live bar chart comparing raw activation distribution vs normalized output
  • Adjustable input variance slider to see normalization effect at different scales
  • Interactive gamma and beta sliders showing how learnable rescaling works

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.