Mixture of Experts (MoE) Architecture - Interactive Visualization

In a Mixture of Experts model, each token is routed to only top-k of N expert FFN layers, making the model sparse - most parameters inactive for any given token. Mixtral 8×7B uses 2 of 8 experts per token, giving GPT-3.5-class quality with ~3× fewer active FLOPs. This demo shows how the router dispatches tokens and tracks expert load.

• Router heatmap - see which experts each token selects and the softmax scores assigned by the gating network
• Expert load tracker - watch how uniform vs skewed routing affects which experts become overloaded
• Active FLOPs calculator - compare total parameters vs active parameters per token at different k and N values
• Load balancing loss - see why auxiliary load balancing loss is necessary to prevent all tokens routing to one expert
• Understand why MoE scales total capacity cheaply but complicates distributed inference across GPUs

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.