EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Model Parallelism
Model Parallelism
7B model · 4 GPUs · Tensor Parallel
GPU Grid (4 GPUs)
GPU 0
Heads 0–7
3.5GB
GPU 1
Heads 8–15
3.5GB
GPU 2
Heads 16–23
3.5GB
GPU 3
Heads 24–31
3.5GB
Memory Per GPU vs Total
Total Model Memory
14 GB
Memory Per GPU
3.5 GB
GPU Efficiency
85%
TP vs PP vs DP Comparison
Tensor (TP)Efficiency: 85% · Mem/GPU: 3.5GB
Pipeline (PP)Efficiency: 76% · Mem/GPU: 3.5GB
Data (DP)Efficiency: 95% · Mem/GPU: 14.0GB
Controls
Number of GPUs
Parallelism Type
Model Size
Tensor Parallel splits each layer across GPUs (low bubble, high comms overhead).

Pipeline Parallel splits layers into stages (low comms, but idle "bubble" time).

Data Parallel replicates the model - each GPU sees a different data shard.

Model Parallelism - Interactive Visualization

Training and serving 70B+ parameter models requires splitting them across multiple GPUs. Tensor parallelism splits individual layers horizontally across GPUs - each GPU handles a slice of attention heads. Pipeline parallelism splits layers into sequential stages - each GPU owns a contiguous block of layers, and micro-batches flow through like a pipeline. Data parallelism replicates the full model and splits the data. Each strategy has different memory, communication, and efficiency trade-offs.

  • GPU grid visualization: see exactly what each GPU holds under TP, PP, DP, or hybrid
  • Pipeline stage animation: watch micro-batches flow through stages, see idle "bubble" time
  • Memory per GPU calculator: compare 7B/13B/70B across 1/2/4/8 GPUs
  • Efficiency comparison bars: TP vs PP vs DP - which wastes the least compute
  • Used in: Megatron-LM, DeepSpeed, FSDP, Ray Train, HuggingFace Accelerate

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.