MoE Router - Expert Selection Heatmap - Interactive Visualization

Mixture-of-Experts routing sends each token to only a small subset of experts (top-K out of N), enabling models with vastly more parameters than a dense model while keeping per-token compute constant. The router is a learned linear layer that produces logits, softmax probabilities, and selects top-K experts. Without load balancing, the router collapses to always routing to the same experts - the auxiliary loss penalizes this imbalance.

• Router produces logits for each expert, softmax gives selection probabilities
• Top-K selection: only K experts process each token, rest are unused
• Expert collapse: router learns to always pick same experts without auxiliary loss
• Capacity factor controls expert buffer size - tokens overflow if experts are overloaded
• Load balancing auxiliary loss encourages uniform utilization across experts

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.