EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Scaling Laws: Compute, Data & Parameters
10^1810^1910^2010^2110^2210^2310^2410^251.522.533.54Compute Budget (FLOPs)LossGPT-2GPT-3ChinchillaLLaMA-7BLLaMA-65BOpt Loss: 1.96Chinchilla-optimal
Optimal N
158.1B
Optimal D
105.4B
N/D ratio
1.500
Est Loss
1.964
Controls
Compute (log)10^23
1e181e25
Model params (log)10^11
1B1T
Overlays
Chinchilla (Hoffmann et al., 2022): for a fixed compute budget, you should scale N and D equally. GPT-3 (175B params, 300B tokens) was significantly undertrained - Chinchilla (70B, 1.4T tokens) matched it at 4× fewer parameters.

Scaling Laws: Compute, Data & Parameters - Interactive Visualization

Scaling laws predict how model loss decreases as you increase compute, parameters, and data. The Chinchilla paper showed that most models were undertrained - optimal training requires N parameters and 20N tokens. This demo lets you explore the optimal allocation of a fixed compute budget.

  • Visualize the power-law relationship between compute budget and achievable loss
  • Find the Chinchilla-optimal parameter count and token count for any FLOPs budget
  • Compare GPT-3 (undertrained) vs Chinchilla (compute-optimal) allocation strategies
  • See how doubling parameters vs doubling data affects final loss differently
  • Understand why training a smaller model on more data often beats a large model trained too briefly

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.