Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/Inference Cost Explorer

Inference Cost Explorer

7B on A100 · batch=8 · ctx=2048

Throughput

1,143 tok/s

Latency/Token

7.0 ms

Cost/1M Tokens

$777.778

Cost Breakdown (Compute / KV Cache / I/O)

35%

50%

15%

Compute (FLOPs): 35%

Memory (KV cache): 50%

I/O bandwidth: 15%

Memory-bound (low arithmetic intensity)

KV cache size: 256 MB for this batch

Batch Size vs Throughput (tokens/s)

optimal

128

Optimal batch size: 128 · Current: 8

Batch Size vs Latency per Token (ms)

128

Larger batches increase latency per request but reduce cost per token

Controls

Batch Size

18128

Context Length

512204816384

Model Size

Hardware

Roofline Model: if arithmetic intensity > ridge point, you're compute-bound. Otherwise memory-bound.

Larger batches improve GPU utilization but increase first-token latency.

Inference Cost Explorer - Interactive Visualization

LLM inference cost breaks down into three components: compute (FLOPs per token), memory bandwidth (loading model weights), and KV cache storage. The roofline model determines whether you are compute-bound or memory-bound - most LLM inference with small batch sizes is memory-bandwidth-limited. Increasing batch size improves GPU utilization but adds latency per request. This explorer lets you find the sweet spot.

Stacked cost breakdown: compute vs KV cache vs memory bandwidth percentage
Batch size vs throughput curve with optimal operating point highlighted
Batch size vs latency curve - see the trade-off in real time
Compare A100 (312 TFLOPS), H100 (990 TFLOPS), RTX 4090 (165 TFLOPS)
Context length slider - see how longer contexts blow up KV cache cost
Cost per 1M tokens for each hardware + model size combination

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.