EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Positional Encoding: Sinusoidal, RoPE & ALiBi
Sinusoidal PE - rows: positions 0–29, cols: dim 0–63
dim →
pos →    Red = +1, Blue = -1
Encoding
Sinusoidal
Used In
BERT, original Transformer
Extrapolation
Poor (fixed range)
How It Works
Uses sin/cos at different frequencies. No learned parameters.
Controls
Encoding Type
Sinusoidal
RoPE
ALiBi
d_model128
64512
n_positions30
10100
Positional encodings tell the model where each token is in the sequence. Transformers have no built-in notion of order - without PE, "cat sat on mat" and "mat on sat cat" look identical.

Positional Encoding: Sinusoidal, RoPE & ALiBi - Interactive Visualization

Positional encoding tells a transformer where each token is in a sequence. This demo compares three approaches: sinusoidal (original Transformer), RoPE (used in LLaMA/Gemini), and ALiBi (used in BLOOM). RoPE enables length generalization by encoding position as rotation angles rather than fixed vectors.

  • Sinusoidal encoding - see how sine and cosine functions at different frequencies encode each position as a unique vector
  • RoPE (Rotary Position Embedding) - visualize how positions are encoded as rotation angles applied to query and key vectors
  • ALiBi - see the linear attention bias that penalizes distant token pairs directly in the attention matrix
  • Compare how each scheme behaves when the sequence length exceeds the training length
  • Understand why RoPE and ALiBi generalize to longer contexts while sinusoidal fails

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.