EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Sparse MoE vs Dense Models - FLOP Comparison
Dense Model
GPT-2 (117M)
FFN Neurons (ALL active per token)
Total params
117M
Active/token
117M
FLOPs/token
234M
2×params
Memory
0.1GB
FP16
Sparse MoE (8 experts, top-2)
936M total, 234M active
FFN Neurons (2/8 experts active per token)
Total params
936M
8× dense
Active/token
234M
2/8 experts
FLOPs/token
200%
of dense
Overflow
15.0%
tokens dropped
FLOPs per Token Comparison
Dense100%
Sparse MoE200%
The MoE tradeoff: more total parameters (knowledge capacity) with fewer active FLOPs per token (compute efficiency). Sparse MoE models can match dense model quality at lower inference cost - but require careful load balancing to avoid wasting capacity.
Controls
Dense Model Size
GPT-2 (117M)
GPT-2 XL (1.5B)
GPT-3 (175B)
Num Experts
Experts8
Top-K
Capacity Factor
Factor1.25
Higher = less overflow, more memory
Key insight: Sparse MoE uses the same FLOPs per token as a dense model that is topK/N fraction of MoE's size - but has the knowledge of the full model.

Sparse MoE vs Dense Models - FLOP Comparison - Interactive Visualization

The key MoE tradeoff: sparse models have far more total parameters than dense models (more knowledge capacity) but use the same FLOPs per token as a dense model that is only topK/N of MoE's size. A 64-expert MoE model with top-2 routing uses the same compute as a dense model 32x smaller - but has 64x more knowledge-storing parameters. Capacity factor controls how much buffer each expert has for tokens beyond its expected load.

  • Sparse MoE: same FLOPs per token as dense, but 10-100x more total parameters
  • Active parameters = topK/N fraction of total - only those experts compute for each token
  • Capacity factor ≥1.0 required to avoid dropping tokens under uneven routing
  • Memory footprint is proportional to total parameters, not active parameters
  • The tradeoff: sparse MoE is compute-efficient but communication-heavy in distributed training

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.