Knowledge Distillation - Interactive Visualization

Knowledge distillation compresses large teacher models into smaller, faster student models by training on soft probability distributions rather than hard labels. Soft labels from a teacher encode inter-class relationships - the model's uncertainty - that one-hot labels discard. The result is a student that punches above its size, retaining most of the teacher's accuracy at a fraction of the inference cost.

• Soft labels: see how teacher output probabilities contain richer signal than hard class labels
• Temperature scaling: higher temperature flattens the distribution, revealing more dark knowledge
• Compression ratio: compare 70B teacher to 7B student - 10x fewer parameters, ~90% accuracy retained
• Latency tradeoff: student models serve 5-10x faster at the same hardware tier

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.