EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Spot Instance Strategy for ML Training
On-Demand Rate
$3.06/hr
Spot Rate
$0.92/hr
Spot Savings
70%
Interrupt Risk
18%/2hr
Expected Interrupts
1
Training Job Timeline - Spot Interruption Scenario
Training
⚡ Interrupt
💾 Ckpt
New
Training
✓ Done
Running
Interrupted
Checkpoint/Complete
Resume overhead
Checkpoint every 500 steps → max lost work on interruption ≈ 500 steps (~0.13 hrs)
Interruption Probability by AZ - A100
us-east-1a
us-east-1b
us-east-1c
p3.16xl
8%
18%
32%
p3.8xl
12%
14%
22%
p2.xl
5%
10%
20%
Low (<12%)
Med (12–22%)
High (>22%)
Cost Calculator - 8hr Job on A100
On-Demand
$24.48
$3.06/hr × 8hr
Spot (with 1 interrupt)
$7.48
$0.92/hr × 8.1hr effective
Net savings: $17.00 (69%) - Spot wins
Spot Controls
Instance Type
Checkpoint Every500 steps
1002000
Job Duration8hr
1hr72hr
Rule of thumb: checkpoint every 15–30 min. Frequent enough to limit lost work; infrequent enough to not tank throughput.

Multi-AZ: run your job across 2–3 AZs simultaneously. If one AZ reclaims your spot, the orchestrator moves the job to another AZ and resumes from last checkpoint.

Spot Instance Strategy for ML Training - Interactive Visualization

Spot and preemptible GPU instances offer the same hardware as on-demand at 60–75% lower cost, with one catch: the cloud provider can reclaim them with 2 minutes of notice. The key to using spot for ML training is an aggressive checkpointing strategy - save model state every 15–30 minutes so an interruption only loses a small amount of compute. With multi-availability-zone fallback, an interrupted job can resume on a new spot instance in under 5 minutes. On an 8-hour A100 training run, spot instances cut cost from $24.48 to $7.36 even accounting for 1–2 interruptions - a 70% net saving.

  • A100 on-demand: $3.06/hr; spot: $0.92/hr - 70% savings with identical hardware performance
  • Checkpoint every 500 steps (~15 min): max lost work on interruption is 15 minutes of compute
  • Multi-AZ fallback reduces effective interruption rate by ~45% by spreading across availability zones
  • Spot wins unless interruption rate exceeds ~30% per 2 hours - watch the heatmap to pick the right AZ

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.