EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/CUDA Programming Model
Total Threads
256
Warps/Block
2
SM Occupancy
13%
L1 Cache Hit
95%
Est. TFLOPS
18.4
GPU Grid - Thread Blocks
Block 0
Block 1
Block 2
Block 3
Click a block to inspect its threads
Memory Hierarchy
Registers
per thread · 256 KB/SM
1 cycle
Shared Memory
per block · 48–96 KB/SM
~32 cycles
L1/Texture Cache
per SM · 128 KB
~80 cycles
L2 Cache
chip-wide · 40 MB
~200 cycles
Global Memory (HBM)
device · 40–80 GB
~600 cycles
Matrix Multiply: Naive vs Shared Memory Tiling
Naive (Global Memory)
1.Each thread loads full row + col from Global Memory
2.Cache miss rate ~80% - DRAM bandwidth bound
3.Throughput: ~4.8 TFLOPS
4.Access pattern: strided → not coalesced
Shared Memory Tiling
1.Block loads tile into fast shared memory (48 KB)
2.All threads reuse tile - 8× fewer DRAM reads
3.Throughput: ~18.2 TFLOPS
4.Coalesced access → 95% L1 hit rate
CUDA Controls
Grid Blocks4
116
Block Dim (N×N)8×8
2×216×16
Warp: 32 threads that execute the same instruction simultaneously. Divergent branches serialize - avoid if/else based on thread ID.

Coalescing: When consecutive threads access consecutive memory addresses, the GPU issues one wide memory transaction instead of N separate ones.

Tile size: Block dim = tile size. 16×16 = 256 threads, 8 warps. Fits nicely in shared mem and achieves ~80% occupancy.

CUDA Programming Model - Interactive Visualization

CUDA organizes GPU execution into a Grid of Thread Blocks, where each Block runs on a single Streaming Multiprocessor (SM) and has exclusive access to fast Shared Memory. The key insight behind efficient GPU kernels is memory locality: naive matrix multiplication reads from slow Global Memory (600-cycle latency) on every access, while tiled shared memory kernels load data once into 32-cycle Shared Memory and reuse it, achieving 3–4× higher effective FLOPS. Warps - groups of 32 threads that execute in lockstep - are the fundamental scheduling unit, and SM occupancy (active warps / max warps) determines how well you hide memory latency.

  • Grid → Blocks → Threads: 3-level hierarchy maps computation to GPU physical SMs and CUDA cores
  • Warp = 32 threads executing same instruction simultaneously - branch divergence serializes execution
  • Shared memory tiling: 16×16 tiles reduce global memory reads by 16× for matrix multiplication
  • Coalesced access: consecutive threads reading consecutive addresses → single wide memory transaction

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.