Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/Training Dynamics

Step

0 / 200

Train Loss

Val Loss

Best Val

Training Controls

Learning rate

LR schedule

Batch size

Key insight:

LR is the most important hyperparameter. Too low: slow convergence. Too high: loss diverges or oscillates.

Warmup ramps LR up gradually - prevents early instability when weights are random.

Larger batch = less gradient noise = smoother loss curve.

Training Dynamics - Interactive Visualization

The training loss curve tells you everything about whether your model is learning correctly. A good run shows steady decrease toward a plateau. A bad run diverges (loss spikes) or plateaus too early. This simulation lets you experiment with learning rate, schedule type, and batch size to see exactly how each hyperparameter affects the loss trajectory.

High LR without warmup: watch the loss spike in early steps then recover (or not)
Warmup + cosine schedule: smooth, stable loss decrease - the modern standard
Large batch size: smoother curve but may require higher LR for the same convergence
Small batch size: noisy curve but often better generalization (regularization effect)
Validation loss vs training loss: see overfitting as the gap opens up
Training dynamics for LLMs: why warmup is critical and why LR matters most

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.