:::tip 🎮 Interactive Playground Visualize this concept: Try the Latency vs Throughput demo on the EngineersOfAI Playground - no code required. :::

Real-Time Inference Design

One Million Requests Per Second, Ten Milliseconds Each

It is 9 AM on a Monday and the ad platform is starting its highest-traffic week of the year. The infrastructure team is watching a dashboard that shows the current state of the ad auction ML system: 1.1 million auction requests per second, 8.2ms median latency, p99 at 9.4ms. The SLA is 10ms end-to-end - including network, feature computation, model inference, and bid scoring.

At 9:07 AM, one of the feature servers starts responding slowly. The Redis lookup for user embedding vectors - normally 0.8ms - climbs to 12ms. The circuit breaker trips at the feature lookup client: instead of waiting for slow Redis responses, requests get the default user embedding vector immediately. Model quality drops slightly (cold-start users get less personalized ads). But latency stays at 8.5ms. Auctions continue.

At 9:12 AM, the Redis cluster recovers. The circuit breaker resets. Full personalization resumes. Total duration of degraded service: 5 minutes. User-visible impact: imperceptible - ads were slightly less personalized but the page still loaded fast. Revenue impact: estimated 0.3% reduction in CTR during those 5 minutes.

Without the circuit breaker: the feature lookup would have blocked for 12ms per request, pushing total latency past 22ms, causing the auction system to time out ML entirely and use static bid prices - estimated 8% revenue impact. The circuit breaker turned a cascade failure into a graceful degradation.

This is the engineering discipline of real-time inference design: not just making the model fast, but designing every interaction between the model and the surrounding system to be explicitly failure-tolerant, latency-bounded, and degradable under pressure.

---

Why This Exists - The Cascade Failure Problem

When ML inference is embedded in a critical path (ad auction, fraud check, real-time pricing), slow ML becomes everyone's problem. A model that takes 200ms instead of 8ms does not just make predictions slower - it blocks the entire request pipeline, causes timeouts, triggers retries that amplify load, and can cascade into a full outage.

The naive architecture is: application calls ML service → waits for result → continues. This works when ML is fast and reliable. It fails catastrophically when:

• Feature retrieval is slow (Redis timeout, network partition)
• The model server is overloaded (GPU OOM, pod restart)
• The ML model itself has a latency regression (new version, different input distribution)
• Dependency chain is unavailable (feature store, embedding service)

Each failure mode needs its own mitigation: circuit breakers for dependency failures, fallback models for service degradation, timeout budgets for latency SLA enforcement, graceful degradation for business continuity.

Real-time inference design is the discipline of engineering all these mitigations before you need them.

---

Historical Context

The circuit breaker pattern for distributed systems was popularized by Michael Nygard's "Release It!" (2007) and Netflix's Hystrix library (2012). The pattern came from electrical engineering: when current exceeds a safe level, the circuit breaker "trips" (opens), stopping current flow to prevent damage. In software: when error rate exceeds a threshold, the circuit breaker "trips", returning fallback responses immediately instead of calling the slow/failing service.

Netflix's 2013 blog post on the Chaos Engineering approach - deliberately injecting failures into production - established that graceful degradation must be tested, not assumed. Their "Chaos Monkey" randomly terminated production services; systems that survived had circuit breakers and fallbacks in place.

For ML systems specifically, the tension between ML quality and SLA compliance became explicit around 2016-2018 as ad tech companies (Google, Facebook, Twitter) published architectures for real-time bidding. The pattern of "fast fallback model" (a lighter model that runs in 2ms) alongside the primary model (running in 8ms) became standard practice. If the primary model cannot be served within the time budget, the fallback model provides a prediction that is slightly less accurate but guaranteed to be fast.

---

The Time Budget Model

Every real-time ML system has a total time budget - the end-to-end latency from request arrival to response. The time budget must be explicitly decomposed and allocated across each stage:

$T_{total} = T_{network\_in} + T_{feature\_lookup} + T_{inference} + T_{postprocess} + T_{network\_out}$

For an ad auction system with 10ms SLA:

Stage	Budget	Notes
Network (client → edge)	1ms	CDN colocation
Feature lookup (Redis)	2ms	P99; use local cache for P50
Model inference (GPU)	4ms	Includes batching wait
Postprocessing	0.5ms	Scoring, format
Network (edge → client)	1ms	CDN colocation
Buffer	1.5ms	For variance
Total	10ms

The buffer is critical - it absorbs natural variance. If you allocate the full 10ms to functional work with zero buffer, any variance in any stage breaches the SLA.

---

Synchronous vs Asynchronous Real-Time Inference

Synchronous Inference

Synchronous inference blocks the request thread until the prediction is returned. The calling service waits.

Best when:

• Prediction is on the critical path (must have it before response)
• Latency budget allows waiting (10ms total, 4ms for inference)
• Request correlation is easy (result must match the request that triggered it)

# synchronous_inference.py - blocking call with timeout budget
import asyncio
import time
from typing import Optional

async def get_prediction_sync(
    request_id: str,
    features: dict,
    ml_client,
    timeout_ms: float = 4.0  # hard deadline from time budget
) -> Optional[dict]:
    """
    Synchronous inference with hard deadline.
    Returns None if timeout is exceeded - caller uses fallback.
    """
    try:
        result = await asyncio.wait_for(
            ml_client.predict(features),
            timeout=timeout_ms / 1000.0
        )
        return result
    except asyncio.TimeoutError:
        # Log timeout - critical for monitoring time budget violations
        prediction_timeouts.labels(reason="inference_timeout").inc()
        return None  # caller will use fallback model
    except Exception as e:
        prediction_errors.labels(error_type=type(e).__name__).inc()
        return None

Asynchronous Inference

Asynchronous inference submits the request to a queue and does not wait for the result. Useful when:

• Prediction is not on the critical path (can be used for later requests)
• Batch processing of events (content moderation, email classification)
• Pre-compute predictions before they are needed (prefetching)

# async_inference.py - fire-and-forget with result store
import asyncio
import json
import redis.asyncio as aioredis
from typing import Callable

class AsyncInferenceClient:
    """
    Submit inference requests asynchronously.
    Results stored in Redis, fetched by request_id when needed.
    """

    def __init__(self, kafka_producer, redis_url: str, ttl_seconds: int = 300):
        self.producer = kafka_producer
        self.redis = aioredis.from_url(redis_url)
        self.ttl = ttl_seconds

    async def submit(self, request_id: str, features: dict) -> str:
        """Submit inference request. Returns request_id for later retrieval."""
        message = {
            "request_id": request_id,
            "features": features,
            "submitted_at": time.time()
        }
        await self.producer.send_and_wait(
            "inference-requests",
            json.dumps(message).encode()
        )
        return request_id

    async def get_result(
        self, request_id: str, max_wait_ms: float = 50
    ) -> Optional[dict]:
        """
        Poll for result. Used for 'best-effort async' pattern:
        submit request, do other work, then check for result.
        """
        deadline = time.perf_counter() + max_wait_ms / 1000
        while time.perf_counter() < deadline:
            raw = await self.redis.get(f"pred:{request_id}")
            if raw:
                return json.loads(raw)
            await asyncio.sleep(0.001)  # 1ms poll interval
        return None  # Use fallback if not ready in time

---

Circuit Breakers for ML Dependencies

The circuit breaker pattern protects the ML serving system from slow or failing upstream dependencies (feature stores, embedding services, external APIs).

# circuit_breaker.py - circuit breaker for ML dependency calls
import asyncio
import time
from enum import Enum
from dataclasses import dataclass, field
from typing import Callable, Any, Optional

class CircuitState(Enum):
    CLOSED = "closed"         # normal - requests pass through
    OPEN = "open"             # tripped - requests get fallback immediately
    HALF_OPEN = "half_open"   # testing - one request let through

@dataclass
class CircuitBreakerConfig:
    failure_threshold: int = 5       # failures before opening
    success_threshold: int = 2       # successes in half-open to close
    timeout_seconds: float = 0.050   # request timeout (50ms)
    reset_timeout_seconds: float = 30.0  # wait before half-open

class MLCircuitBreaker:
    """
    Circuit breaker for ML inference dependency calls.
    Protects against slow feature stores, Redis timeouts, etc.
    """

    def __init__(self, name: str, config: CircuitBreakerConfig = None):
        self.name = name
        self.config = config or CircuitBreakerConfig()
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time: Optional[float] = None
        self._lock = asyncio.Lock()

    async def call(
        self,
        fn: Callable,
        *args,
        fallback: Any = None,
        **kwargs
    ) -> Any:
        """
        Execute fn through circuit breaker.
        Returns fallback if circuit is open or call fails.
        """
        async with self._lock:
            state = await self._get_state()

        if state == CircuitState.OPEN:
            circuit_breaker_rejections.labels(breaker=self.name).inc()
            return fallback

        try:
            result = await asyncio.wait_for(
                fn(*args, **kwargs),
                timeout=self.config.timeout_seconds
            )
            await self._record_success()
            return result

        except (asyncio.TimeoutError, Exception) as e:
            await self._record_failure(e)
            return fallback

    async def _get_state(self) -> CircuitState:
        """Check if OPEN circuit should transition to HALF_OPEN."""
        if self.state == CircuitState.OPEN:
            if (time.monotonic() - self.last_failure_time
                    > self.config.reset_timeout_seconds):
                self.state = CircuitState.HALF_OPEN
                self.success_count = 0
        return self.state

    async def _record_success(self):
        async with self._lock:
            if self.state == CircuitState.HALF_OPEN:
                self.success_count += 1
                if self.success_count >= self.config.success_threshold:
                    self.state = CircuitState.CLOSED
                    self.failure_count = 0
                    print(f"Circuit {self.name}: CLOSED (recovered)")
            elif self.state == CircuitState.CLOSED:
                self.failure_count = max(0, self.failure_count - 1)

    async def _record_failure(self, error: Exception):
        async with self._lock:
            self.failure_count += 1
            self.last_failure_time = time.monotonic()
            if self.failure_count >= self.config.failure_threshold:
                if self.state != CircuitState.OPEN:
                    self.state = CircuitState.OPEN
                    print(f"Circuit {self.name}: OPEN (tripped by {error})")


# Usage in ad auction serving
feature_circuit = MLCircuitBreaker("redis_features")
embedding_circuit = MLCircuitBreaker("embedding_service")

async def get_auction_features(user_id: str, item_id: str) -> dict:
    """Fetch auction features with circuit breaker protection."""
    # User embedding - falls back to global average embedding
    user_embedding = await feature_circuit.call(
        redis_client.get_user_embedding,
        user_id,
        fallback=DEFAULT_USER_EMBEDDING
    )

    # Item context - falls back to item category average
    item_features = await embedding_circuit.call(
        embedding_service.get_item_features,
        item_id,
        fallback={"category_embedding": DEFAULT_CATEGORY_EMBEDDING}
    )

    return {"user_embedding": user_embedding, **item_features}

---

Fallback Model Architecture

When the primary model is unavailable or too slow, a fallback model provides predictions with lower quality but guaranteed latency.

# fallback_model.py - tiered model serving with graceful degradation
import asyncio
import time
from typing import Tuple

class TieredModelServer:
    """
    Serves predictions through a hierarchy of models:
    1. Primary model (best quality, 4ms target)
    2. Secondary model (good quality, 1ms target)
    3. Static rules / heuristics (0ms, always available)
    """

    def __init__(self, primary_model, secondary_model):
        self.primary = primary_model
        self.secondary = secondary_model
        self.primary_circuit = MLCircuitBreaker("primary_model")
        self.secondary_circuit = MLCircuitBreaker("secondary_model")

    async def predict(
        self, features: dict, time_budget_ms: float = 4.0
    ) -> Tuple[dict, str]:
        """
        Returns (prediction, source) where source indicates which tier served.
        """
        deadline = time.perf_counter() + time_budget_ms / 1000

        # Tier 1: Primary model
        remaining_ms = (deadline - time.perf_counter()) * 1000
        if remaining_ms > 2.0:  # only try if enough budget remains
            result = await self.primary_circuit.call(
                self.primary.predict,
                features,
                fallback=None
            )
            if result is not None:
                return result, "primary"

        # Tier 2: Secondary (smaller/faster) model
        remaining_ms = (deadline - time.perf_counter()) * 1000
        if remaining_ms > 0.5:
            result = await self.secondary_circuit.call(
                self.secondary.predict,
                features,
                fallback=None
            )
            if result is not None:
                return result, "secondary"

        # Tier 3: Static fallback (always fast, lowest quality)
        return self._static_fallback(features), "fallback"

    def _static_fallback(self, features: dict) -> dict:
        """
        Rule-based fallback that runs in microseconds.
        For ad auction: return category-average CTR estimate.
        """
        category = features.get("item_category", "unknown")
        return {
            "score": CATEGORY_AVERAGE_SCORES.get(category, 0.05),
            "confidence": "low",
        }

---

Request Lifecycle Design for 1M QPS

At 1M QPS, every architectural decision multiplies. A 1μs overhead per request is 1 second of CPU time per second - at 100 cores, that is 1% CPU utilization from a single 1μs overhead.

Key design principles for this architecture:

Parallel feature fetch: all feature lookups (user, item, context) start simultaneously. The request waits for all of them, or uses circuit-broken fallbacks for slow ones. Serial feature lookup multiplies latency; parallel limits it to the slowest single lookup.

Per-request deadline propagation: every downstream call receives the remaining time budget as a deadline. Feature fetcher gets 2ms. Model inference gets 4ms. If feature fetch takes 3ms and exceeds budget, the model call gets 0ms - skip model, use fallback immediately. Deadline propagation prevents a single slow stage from consuming time meant for later stages.

# deadline_propagation.py - per-request deadline propagation
import asyncio
import time
from dataclasses import dataclass

@dataclass
class RequestContext:
    request_id: str
    deadline: float  # absolute time.perf_counter() value

    @classmethod
    def with_budget_ms(cls, request_id: str, budget_ms: float) -> "RequestContext":
        return cls(
            request_id=request_id,
            deadline=time.perf_counter() + budget_ms / 1000.0
        )

    @property
    def remaining_ms(self) -> float:
        return max(0, (self.deadline - time.perf_counter()) * 1000)

    @property
    def is_expired(self) -> bool:
        return time.perf_counter() >= self.deadline

    def child_timeout_s(self, max_fraction: float = 1.0) -> float:
        """Return timeout for a child operation (max fraction of remaining budget)."""
        return min(self.remaining_ms * max_fraction / 1000, 30.0)


async def handle_auction_request(request, ctx: RequestContext) -> dict:
    """Full auction pipeline with deadline propagation."""
    if ctx.is_expired:
        return static_fallback(request)

    # Stage 1: Parallel feature fetch (up to 40% of budget)
    feature_timeout = ctx.child_timeout_s(max_fraction=0.4)
    try:
        user_feat, item_feat = await asyncio.wait_for(
            asyncio.gather(
                get_user_features(request.user_id),
                get_item_features(request.item_id),
                return_exceptions=True
            ),
            timeout=feature_timeout
        )
    except asyncio.TimeoutError:
        user_feat = DEFAULT_USER_EMBEDDING
        item_feat = DEFAULT_ITEM_EMBEDDING

    # Replace exceptions with defaults
    if isinstance(user_feat, Exception):
        user_feat = DEFAULT_USER_EMBEDDING
    if isinstance(item_feat, Exception):
        item_feat = DEFAULT_ITEM_EMBEDDING

    if ctx.is_expired:
        return score_with_features(user_feat, item_feat, method="lightweight")

    # Stage 2: Model inference (remaining budget)
    inference_timeout = ctx.child_timeout_s()
    try:
        score = await asyncio.wait_for(
            model_server.predict({"user": user_feat, "item": item_feat}),
            timeout=inference_timeout
        )
        return {"score": score, "method": "primary_model"}
    except asyncio.TimeoutError:
        return score_with_features(user_feat, item_feat, method="fallback")

---

Timeout Budget Management in Practice

A common mistake is using fixed timeouts throughout the stack. If the total SLA is 10ms and you set 5ms timeouts everywhere, a request that uses 4ms at the edge, 5ms at feature fetch, and 5ms at inference takes 14ms total - 40% over SLA - even though no individual component exceeded its timeout.

Deadline-based timeouts (as shown above) solve this: each stage gets the remaining budget, not a fixed budget. The total latency can never exceed the original deadline regardless of how stages distribute time.

# timeout_math.py - why fixed timeouts fail, deadline propagation works
import asyncio
import time

async def fixed_timeout_example(total_budget_ms=10):
    """Fixed timeouts: stages can independently use full budget."""
    timeout_per_stage = 5  # ms - seems conservative

    start = time.perf_counter()
    # Stage 1 uses 4ms (within 5ms timeout) ✓
    await asyncio.sleep(0.004)
    # Stage 2 uses 5ms (within 5ms timeout) ✓
    await asyncio.sleep(0.005)
    # Stage 3 uses 4ms (within 5ms timeout) ✓
    await asyncio.sleep(0.004)
    total = (time.perf_counter() - start) * 1000
    print(f"Fixed timeout total: {total:.1f}ms")  # 13ms - SLA breached!


async def deadline_timeout_example(total_budget_ms=10):
    """Deadline propagation: total time is hard-bounded."""
    deadline = time.perf_counter() + total_budget_ms / 1000

    start = time.perf_counter()
    # Stage 1: 40% budget = 4ms
    remaining = max(0, (deadline - time.perf_counter()) * 1000)
    await asyncio.wait_for(asyncio.sleep(0.004), timeout=remaining * 0.4 / 1000)
    # Stage 2: remaining budget after stage 1
    remaining = max(0, (deadline - time.perf_counter()) * 1000)
    await asyncio.wait_for(asyncio.sleep(0.003), timeout=remaining / 1000)
    # Stage 3: whatever is left
    remaining = max(0, (deadline - time.perf_counter()) * 1000)
    await asyncio.wait_for(asyncio.sleep(0.002), timeout=remaining / 1000)

    total = (time.perf_counter() - start) * 1000
    print(f"Deadline total: {total:.1f}ms")  # ≤10ms guaranteed

---

Production Engineering Notes

Traffic Shaping at the Edge

For 1M QPS, traffic shaping before requests reach the ML stack prevents overload:

Rate limiting: per-user, per-app, per-IP rate limits prevent any single client from saturating capacity. Use token bucket algorithm (allows bursts up to N× average).

Priority queues: high-value auction requests (premium advertisers, high CPM) get priority over lower-value requests when the system is under load. Drop lower-priority requests first during overload rather than degrading uniformly.

Load shedding: if the inference queue depth exceeds a threshold, start returning 503 or static fallback responses for low-priority requests instead of queuing them and increasing latency for everyone.

SLA Monitoring for Real-Time Inference

At 1M QPS, even 0.1% of requests is 1,000 per second. Define SLA success as: p99 of end-to-end latency below threshold, measured over rolling 1-minute windows. Alert on p99 breach, not mean latency breach.

---

Common Mistakes

:::danger Not Propagating Deadlines Across Service Boundaries When an upstream service calls your ML endpoint with a 10ms remaining budget, that budget context must propagate to every downstream call you make. If you use a fixed 30-second gRPC deadline for feature lookup, you will wait 30 seconds for a timeout even though the upstream has already failed after 10ms. Every inter-service call must receive the deadline from the incoming request context, transmitted via gRPC metadata, HTTP header, or request payload. :::

:::danger Retrying on Timeout in Real-Time Serving Retrying a failed ML request in a 10ms SLA context almost never helps and usually makes things worse. If the first attempt timed out at 4ms, there is no time for a retry. Retries amplify load on an already-stressed system, accelerating cascade failures. In real-time serving, use fallbacks - not retries. Retries belong in batch jobs where latency is not the constraint. :::

:::warning Cold-Start Latency for Circuit Half-Open State When a circuit breaker transitions from OPEN to HALF-OPEN to test recovery, the first request goes through normally. If the system has recovered and is now healthy, that request returns in normal time. But if you are testing recovery too aggressively - resetting after only 5 seconds of downtime - the service may not be fully warmed up and the first few requests will be slow, retripping the circuit. Set reset_timeout_seconds long enough for the underlying service to fully recover (typically 30-60 seconds for ML services with GPU warm-up requirements). :::

---

Interview Q&A

Q1: How do you design a real-time ML serving system for 1M QPS with a 10ms SLA?

A: Start by decomposing the time budget explicitly: network 2ms, feature fetch 2ms (parallel), model inference 4ms, buffer 2ms. Then design each stage to be independently latency-bounded. Feature fetch uses circuit breakers so slow Redis does not exceed 2ms budget. Model inference uses dynamic batching with max_wait_ms calibrated to the latency budget minus current elapsed time. Use deadline propagation: each downstream call receives the remaining budget, not a fixed timeout. Horizontally scale the feature layer (Redis cluster, consistent hashing for locality), inference layer (GPU pods with KEDA), and edge layer (stateless, CDN colocation near user). At 1M QPS, the feature fetch layer handles 2M+ Redis lookups per second - require Redis clusters in each region with read replicas, and maintain a small in-process LRU cache (last 10K user/item embeddings) to absorb the hottest keys without touching Redis.

Q2: What is the circuit breaker pattern and how does it apply to ML serving?

A: The circuit breaker pattern prevents cascade failures by short-circuiting calls to failing dependencies. In ML serving, every dependency (feature store, embedding service, model server) can fail or become slow. Without circuit breakers, slow dependencies block request threads, causing thread pool exhaustion, request queue buildup, and eventually system-wide timeout. The circuit breaker has three states: CLOSED (normal - calls pass through), OPEN (tripped - calls return fallback immediately), HALF_OPEN (testing - one request let through). In CLOSED state, if failures exceed a threshold (5 failures in 30 seconds), it trips to OPEN. After a reset timeout (30s-1min), it moves to HALF_OPEN to test if the dependency recovered. Successful recovery closes the circuit again. For ML: feature lookup failures trigger fallback to default embeddings; model server failures trigger fallback to the secondary model; complete inference failures trigger static rule-based scoring.

Q3: What is a fallback model and when should you use it?

A: A fallback model is a simpler, faster model that can serve predictions when the primary model is unavailable or too slow. The typical hierarchy is: primary model (complex, best quality, 4ms latency) → secondary model (simpler, good quality, 0.5ms latency) → static rules (zero latency, lowest quality). Use the fallback when: (a) the primary model's circuit breaker is open; (b) the remaining time budget is insufficient for the primary model's typical latency; (c) the model server returns an error; or (d) feature retrieval failed and the primary model cannot produce quality predictions on incomplete features. The fallback model should be: always loaded in memory (never cold-started), designed to work with minimal features (does not depend on the features that might be unavailable), and explicitly tested at the same quality bar as the primary model to understand the accuracy tradeoff.

Q4: How do you prevent cascade failures in a real-time ML serving system under load?

A: Four mechanisms work together. (1) Circuit breakers: as described above, short-circuit failing dependencies to prevent backpressure from one slow service taking down others. (2) Load shedding: when the inference queue depth exceeds a threshold, start rejecting low-priority requests with 503 and returning static scores, rather than letting the queue grow indefinitely and causing latency to explode for all requests. (3) Bulkheads: separate thread pools / connection pools for different priority tiers so that a surge of low-priority requests cannot exhaust the resources needed for high-priority requests. (4) Timeout cascading: every call has a deadline derived from the incoming request's remaining budget. A request that is about to time out at the upstream SLA has zero time budget for downstream calls - it should return immediately from fallback. These mechanisms together ensure that a failure in one component degrades gracefully rather than cascading.

Q5: How do you measure and enforce a 10ms p99 SLA for real-time ML serving?

A: SLA enforcement requires measurement, alerting, and architectural safeguards. Measurement: instrument every stage with distributed tracing (OpenTelemetry spans) and record end-to-end latency as a Prometheus histogram with fine-grained buckets (1ms, 2ms, 3ms, 5ms, 7ms, 10ms, 15ms, 25ms). Monitor p99 as histogram_quantile(0.99, ...) over rolling 1-minute windows. Alerting: page on-call if p99 exceeds 10ms for more than 2 consecutive minutes. Warning if p95 exceeds 8ms (early signal). Architectural enforcement: set deadline-based timeouts that sum to less than 10ms (with buffer) - the system physically cannot take longer than the deadline. Load shedding ensures the system returns a fast fallback rather than queuing when overloaded. Test the SLA under failure scenarios (Redis slow, GPU OOM) with chaos engineering - verify that fallback paths respect the 10ms budget.