Context Extension - YaRN, ALiBi, LongRoPE Comparison - Interactive Visualization

Language models trained with RoPE (Rotary Position Embeddings) see catastrophic perplexity degradation when deployed beyond their trained context length - the perplexity cliff. Extension methods like YaRN (NTK-aware interpolation), ALiBi (linear attention biases), and LongRoPE (non-uniform scaling) push this cliff further out. YaRN applies different scaling factors to different frequency bands: high-frequency dimensions are interpolated, low-frequency dimensions are extrapolated.

• RoPE encodes position via rotation in frequency space - frequencies from θ_i = base^(-2i/d)
• At context > trained limit: rotation angles unseen during training → perplexity spikes
• YaRN: NTK-aware - scales base frequency by factor s, enabling 4-32x context extension
• ALiBi: no positional embeddings at all - adds linear bias to attention scores by distance
• LongRoPE: non-uniform scaling - different scale factors per frequency dimension

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.