EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/BERT Masked Language Modeling (MLM)
BERT Masked Language Modeling
Click any token to mask it. BERT predicts the masked token by attending to all surrounding tokens bidirectionally.
Sentence tokens (click to mask)
The
[MASK]
brown
fox
[MASK]
over
the
lazy
dog
2 tokens masked. Click a masked token to see its predictions.
Bidirectional attention to [MASK] at position 1 ("quick")
The
brown
fox
jumps
over
the
lazy
dog
Darker = higher attention weight. BERT attends bidirectionally - both left and right context.
Top-5 predictions for [MASK] at position 1
1"quick"✓ correct52.0%
2"fast"22.0%
3"brown"9.0%
4"big"8.0%
5"slow"4.0%
Controls
Preset sentence
MLM training: 15% of tokens are masked. BERT predicts them using bidirectional context - unlike GPT which only sees left context.
15% masking breakdown:
80% → [MASK]
10% → random token
10% → unchanged
This prevents the model from only learning [MASK] positions.

BERT Masked Language Modeling (MLM) - Interactive Visualization

BERT (Devlin et al., 2018) introduced Masked Language Modeling as a pretraining objective where 15% of input tokens are randomly masked and the model must predict them using bidirectional context. Unlike GPT's left-to-right prediction, BERT attends to both past and future tokens simultaneously, making it powerful for understanding tasks like classification and NER.

  • Click any token in a sentence to mask it and see BERT's top-5 predictions with probability bars
  • Bidirectional attention heatmap showing which surrounding tokens influence the masked prediction
  • Four preset sentences covering different vocabulary types and prediction difficulties
  • 15% masking breakdown: 80% replaced with [MASK], 10% random, 10% unchanged

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.