EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Gradient Boosting & Residuals
Gradient Boosting
Sequentially fit residuals with weak learners
# Estimators
rounds30
1100
Learning Rate
η0.20
0.011.0
Stats
Round0
Train MSE0.07986
Residual RMS0.2826
Red lines = residuals (gaps between prediction and truth).

Each round trains a stump on the residuals, then adds its prediction × η.

Low η = slow but generalises better. High η = fast but may overfit.

Gradient Boosting & Residuals - Interactive Visualization

Gradient boosting builds an ensemble sequentially: each new tree is trained to predict the residual errors (negative gradients of the loss) that the current ensemble makes. By repeatedly correcting mistakes, even very shallow trees stack up into a powerful predictor. XGBoost, LightGBM, and CatBoost are all variants of this core idea.

  • See the residuals shrink round by round as each new tree corrects the remaining errors
  • Understand how the learning rate (shrinkage) controls the contribution of each tree - smaller rate needs more rounds but generalizes better
  • Watch the ensemble prediction curve fit increasingly complex patterns as more trees are added
  • Learn the difference between AdaBoost (re-weight samples) and gradient boosting (fit residuals)
  • See why early stopping is essential: boosting can overfit if you run too many rounds

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.