Language Modeling: MLM vs CLM - Interactive Visualization

Two paradigms power modern LLMs: masked language modeling (BERT - predict [MASK] tokens) and causal language modeling (GPT - predict the next token). Each shapes the model's strengths: BERT excels at understanding, GPT at generation. This demo lets you see both applied to the same sentence.

• MLM (BERT-style) - see how random tokens are masked and the model predicts each one using both left and right context
• CLM (GPT-style) - see how the model autoregressively predicts each next token using only past context
• Enter any sentence and compare which tokens are hardest to predict under each objective
• Understand why bidirectional context (BERT) makes better embeddings but cannot generate text
• See how T5-style span masking sits between MLM and CLM as a sequence-to-sequence pretraining objective

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.