Model Soup - Weight Averaging Ensemble - Interactive Visualization

Model soup (Wortsman et al. 2022) discovered that averaging the weights of multiple independently fine-tuned checkpoints consistently outperforms any single checkpoint. The key insight: models fine-tuned from the same pretrained weights lie in the same loss basin, so averaging their weights stays in that basin rather than jumping to a high-loss region. This is fundamentally different from output ensemble (averaging predictions) - it is weight-space averaging.

• Average weights of N fine-tuned checkpoints - not predictions, actual model weights
• Works because all checkpoints start from same pretrained init - same loss basin
• Uniform averaging: equal weight to all checkpoints, simple and often best
• Performance-weighted: weight proportional to validation accuracy
• Consistently adds 0.5-2% accuracy over best single model with zero inference cost

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.