Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/GPU Cluster Scheduling

GPU Cluster (8 GPUs)

GPU 0 (A100 40GB)

Util82%

VRAM38/40GB

GPT-fine-tune

GPU 1 (A100 40GB)

Util65%

VRAM28/40GB

BERT-train

GPU 2 (A100 40GB)

Util45%

VRAM20/40GB

ResNet-eval

GPU 3 (A100 40GB)

Util20%

VRAM8/40GB

XGB-sweep

GPU 4 (A100 40GB)

Util0%

VRAM0/40GB

idle

GPU 5 (A100 40GB)

Util0%

VRAM0/40GB

idle

GPU 6 (A100 80GB)

Util78%

VRAM34/80GB

Stable-Diff

GPU 7 (A100 80GB)

Util0%

VRAM0/80GB

idle

Running Jobs

Job	Priority	GPU Mem	Util %	Duration	GPU
GPT-fine-tune	high	38GB	82%	4h	GPU 0
BERT-train	high	28GB	65%	2h	GPU 1
ResNet-eval	medium	20GB	45%	30m	GPU 2
XGB-sweep	low	8GB	20%	1h	GPU 3
Stable-Diff	high	34GB	78%	6h	GPU 6

Job Queue (0 waiting)

Queue empty - add jobs above

Scheduler Controls

Scheduling Strategy

Pack onto busiest GPU - saves power, causes hot-spots

Show Memory Fragmentation

GPU scheduling must balance utilization, memory capacity, and job priority.

Bin-packing maximizes utilization per node, reducing idle GPUs. Spread reduces noisy-neighbor effects in shared clusters.

GPU Cluster Scheduling - Interactive Visualization

GPU clusters for ML training require careful scheduling: each GPU has fixed VRAM, and over-subscribing causes OOM errors. Jobs have different priorities (high-priority fine-tuning vs low-priority batch eval) and different VRAM requirements (4GB for small evals to 38GB for LLM fine-tuning on A100s). Bin-packing schedulers pack jobs onto the fewest GPUs to reduce idle hardware costs; spread schedulers distribute jobs to reduce interference between tenants. This demo simulates an 8-GPU cluster with a priority queue and live job assignment.

Bin-packing: assign each new job to the busiest GPU that still has capacity - maximizes utilization, risks hot-spots
Spread: assign to the least-busy GPU - reduces noisy-neighbor interference, leaves GPUs partially idle
Priority queue: high-priority jobs (fine-tuning, production training) preempt low-priority jobs (batch eval)
VRAM management: each GPU has a fixed memory budget - scheduling must account for both compute and memory
Memory fragmentation: small jobs can fragment a GPU leaving insufficient contiguous memory for large models
A100 80GB GPUs handle full LLM fine-tuning; V100 16GB GPUs handle smaller models and eval workloads

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.