Must retrain batch layer on full history. Takes hours to days. Speed layer continues with old model in meantime.
Kappa approach
Replay historical topic through new consumer group. Single pipeline handles both past and present.
Tradeoff Comparison
Lambda
Kappa
Code complexity
High (2 pipelines)
Lower (1 pipeline)
Result accuracy
Exact (batch corrects)
Exact (stream corrects)
Data freshness
Lag = batch window (24h)
Near real-time always
Reprocessing time
24h per batch cycle
~9.6h (stream replay)
Operational cost
Higher (two systems)
Lower (one system)
Fault tolerance
High (batch is source of truth)
Depends on log retention
Architecture Controls
Scenario
Batch Window Size
Display
Kappa won for most modern ML systems - managing two separate code paths (Lambda) leads to divergence bugs.
Use Lambda only when you need guaranteed historical correction windows and your stream processor lacks replay.
Lambda vs Kappa Architecture - Interactive Visualization
Lambda Architecture splits the pipeline into three layers: a batch layer that periodically recomputes from raw historical data, a speed layer that handles recent events in real-time, and a serving layer that merges both views. Kappa Architecture eliminates the batch layer entirely - a single stream processor handles both real-time events and historical replay. For most modern ML systems, Kappa wins: maintaining two separate code paths in Lambda inevitably leads to subtle divergence between batch and speed results, doubling operational complexity. The only case for Lambda is when your stream processor cannot efficiently replay large historical datasets.
Lambda batch layer: recomputes features/predictions from scratch on a schedule - accurate but adds a freshness lag equal to the batch window
Speed layer: processes only recent events - fast but approximate; can have different logic than the batch layer
Kappa: replay old Kafka events through the same consumer group as a new consumer group - one code path handles all time ranges
Divergence risk in Lambda: any bug in the speed layer logic that differs from the batch layer produces inconsistent predictions
Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.
