EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Kafka Architecture
Messages Produced
0
Throughput /s
0
Partitions
3
Consumer Lag
3
Producers
App Server
Microservice
CDC
Topic: user-events (RF=2)
Partition 0offset: 2
user_login
purchase
page_view
HEAD→3
← Consumer A-1
Partition 1offset: 5
purchase
page_view
error_log
HEAD→6
← Consumer A-2
Partition 2offset: 8
page_view
error_log
signup
HEAD→9
← Consumer A-3
Each partition replicated 2x across brokers - 1 follower(s) per partition
Consumer Groups
Group A (balanced)
P0 → Consumer A-1
P1 → Consumer A-2
P2 → Consumer A-3
Group B (single)
B-1 reads all 3 partitions
Bottleneck - 1 consumer, 3 partitions
Kafka rule: Each partition is consumed by at most one consumer per group. Group A (3 consumers, 3 partitions) has perfect parallelism. Group B (1 consumer) reads all partitions serially - throughput capped at 1x. Adding more consumers than partitions wastes resources - idle consumers.
Kafka Config
Partitions3
16
Replication Factor2
13 (HA)
Highlight Group
Key Rule
Max parallelism = partition count. 3 partitions = max 3 consumers working in parallel per group.
Retention keeps messages for all consumer groups independently. Group B can replay from offset 0 without affecting Group A - durable pub/sub at scale.

Kafka Architecture - Interactive Visualization

Apache Kafka is the backbone of real-time data infrastructure at Netflix, Uber, LinkedIn, and most ML feature pipelines. Its architecture separates producers from consumers through a partitioned, replicated log. Understanding how partitions distribute load, how consumer groups enable parallel reads, and how offsets track exactly-once delivery is essential for building reliable streaming ML systems. This demo makes the internals visible.

  • Topics are split into partitions - each partition is an ordered, immutable log with a numeric offset
  • Producers append to partitions using a key hash or round-robin; ordering is guaranteed within a partition
  • Consumer groups allow parallel reads - each partition is assigned to exactly one consumer in a group
  • Replication factor (usually 3) means each partition has one leader and N-1 followers for fault tolerance
  • Consumer lag = latest offset − committed offset; high lag means consumers are falling behind producers
  • Used in production for ML feature pipelines, real-time scoring, event sourcing, and CDC (change data capture)

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.