BPE Tokenisation - Interactive Visualization

Byte Pair Encoding (BPE) is the tokenization algorithm used by GPT-2, GPT-3, GPT-4, and most modern LLMs. It starts with individual characters and repeatedly merges the most frequent adjacent pair into a single token - finding the most data-efficient vocabulary. This step-by-step visualizer shows every merge operation and colors tokens by their depth (how many merges created them).

• Watch the BPE merge algorithm operate one step at a time on your own text
• Token colors show merge depth: gray=characters, indigo=merged tokens, purple=deeply merged
• See compression ratio: how many fewer tokens than characters after BPE
• Rare words and unknown words split into recognizable subword pieces - no out-of-vocabulary
• Adjust max merges to find the vocabulary size tradeoff point
• Used by: GPT-4, Claude, Llama, Mistral, and virtually all modern LLMs

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.