EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Spark Batch Processing Visualizer
Spark Job Execution Visualizer
Feature Generation Pipeline - 32 partitions, 2 shuffles
Spark DAG - Operations
TRANSFORM
read_parquet()
32p · 500M
→
TRANSFORM
filter()
32p · 400M
→
SHUFFLE
TRANSFORM
groupBy()
200p · 400M
shuffle
→
TRANSFORM
agg(sum, count)
32p · 10M
→
SHUFFLE
TRANSFORM
join(user_table)
200p · 10M
shuffle
→
ACTION
write.parquet()
32p · 10M
Physical Plan - Stages (separated by shuffle boundaries)
Stage 1
read_parquet()
filter()
groupBy()
32 tasks
Stage 2
agg(sum, count)
join(user_table)
32 tasks
Stage 3
write.parquet()
32 tasks
Cluster Executor View
Executor 1
task
task
task
Executor 2
task
task
task
Executor 3
task
task
task
Executor 4
task
task
task
Stage 1 of 3 - Est. 1 min job time
Job Type
Configuration
Partitions32
Options
Job Stats
Stages: 3
Shuffles: 2
Partitions: 32
Executors: 4
Performance Tip
Shuffles are the most expensive operations. Every shuffle writes data to disk and transfers across network. Minimize shuffles by filtering early and caching hot datasets.

Spark Batch Processing Visualizer - Interactive Visualization

Apache Spark executes ML feature pipelines as a directed acyclic graph (DAG) of operations. Transformations are lazy - they build a logical plan. Actions trigger execution. Shuffle operations (groupBy, join) split the DAG into stages separated by network data transfer across executors. Each stage runs tasks in parallel across the cluster. Understanding stage boundaries, partition count, and shuffle cost is essential for optimizing Spark ML pipelines. Caching intermediate RDDs eliminates redundant recomputation for iterative algorithms.

  • Narrow transformations (filter, map) stay within partitions - no network I/O, run in the same stage
  • Wide transformations (groupBy, join) shuffle data across the network - each creates a new stage boundary
  • Partition count: aim for 2–4 partitions per CPU core; too few wastes compute, too many adds overhead
  • Caching: store intermediate results in memory to skip re-reading Parquet on subsequent actions in the same job

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.