LLM Gateway and Routing

The Night Everything Stopped

It was 2:47 AM when the on-call engineer's phone lit up. Every single LLM-powered feature in the product had returned errors for the past eleven minutes. The customer support chatbot: down. The document summarizer: down. The search assistant: down. All three features shared a single dependency - direct API calls to OpenAI. And at 2:36 AM, OpenAI had started returning HTTP 429 errors for their entire API surface due to a capacity incident.

The engineering team had built three features, each making direct API calls with its own API key, its own retry logic (or lack thereof), and its own error handling. When the incident ended at 3:08 AM, they had thirty-two minutes of complete downtime logged. The postmortem the next morning had a clear root cause: no fallback, no routing, no circuit breaking. Every egg in one basket.

The proposed fix started as "we should add retry logic to each service." But the team quickly realized that was the wrong abstraction. They needed a shared infrastructure layer that all LLM features routed through - a gateway that could handle provider failures, route between models, track costs, and enforce rate limits in one place. Rebuilding retry logic in every service was not an option they were willing to maintain.

They deployed LiteLLM Proxy in front of their LLM traffic. It took three days of engineering. Two weeks later, when OpenAI had another brief degradation (15 minutes, this time), the monitoring showed 11 requests failed before the circuit breaker tripped and traffic automatically shifted to Anthropic Claude. User-visible error count: zero. The on-call engineer slept through it.

This is the central argument for an LLM gateway: LLM providers are external dependencies with real SLAs below 100%, and your product needs a reliability layer between your application code and the provider. A gateway is not complexity for its own sake - it is the difference between "we were down for 32 minutes" and "users never noticed."

Why This Exists

When you have one LLM feature calling one provider, direct API calls are fine. Add a second feature, a third, different models for different tasks, a cost cap per user, a staging environment that should not call expensive models, and a fallback for when the primary provider goes down - suddenly you have eight different services all solving the same infrastructure problems independently.

An LLM gateway centralizes:

• Authentication: one place to manage API keys for all providers
• Rate limiting: prevent individual users or features from burning through your quota
• Cost tracking: per-user, per-feature, per-experiment attribution
• Routing: send different request types to different models
• Fallbacks: automatic failover when a provider is degraded
• Caching: deduplicate identical requests across all services
• Logging: unified request log for debugging and compliance
• Model version pinning: prevent silent behavior changes from model updates

The alternative - each service managing all of this independently - leads to inconsistent behavior, duplicated code, and a cross-cutting concern spread across every team.

What an LLM Gateway Looks Like

Open-Source Gateway Comparison

Feature	LiteLLM	PortKey	OpenRouter
Deployment	Self-hosted or cloud	Cloud (OSS core)	Cloud
Models supported	100+	100+	200+
Load balancing	Yes	Yes	Limited
Budget management	Yes	Yes	No
Semantic caching	Yes	Yes	No
Self-hostable	Yes (Docker)	Partial	No
Fallback chains	Yes	Yes	Yes
SDK changes required	No (OpenAI-compatible)	No	No
Best for	Internal infra teams	Teams wanting managed	Quick provider switching

LiteLLM is the default recommendation for teams that want to self-host. It exposes an OpenAI-compatible proxy that accepts any OpenAI SDK call and routes to the configured provider. Your application code does not change - you only change the base_url to point at your LiteLLM proxy.

PortKey offers a similar feature set with a hosted option and a more polished dashboard. Good choice if you do not want to maintain the infrastructure.

OpenRouter is a hosted service that gives you access to 200+ models through a single API key. No self-hosting required. Limited to routing and model access - no budget management, no custom routing logic.

LiteLLM Deep Dive

Basic Setup

# litellm_config.yaml
model_list:
  # Primary: OpenAI GPT-4o
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  # Secondary: Claude 3.5 Sonnet (fallback for gpt-4o)
  - model_name: gpt-4o
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

  # Fast/cheap tier
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  # Fast/cheap fallback
  - model_name: gpt-4o-mini
    litellm_params:
      model: anthropic/claude-3-haiku-20240307
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: usage-based-routing
  enable_pre_call_checks: true
  num_retries: 3
  timeout: 30

  # Circuit breaker: disable a model after 3 failures in 60s
  allowed_fails: 3
  cooldown_time: 60  # seconds before re-enabling a failed model

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

# Start LiteLLM proxy
docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -e LITELLM_MASTER_KEY=$LITELLM_MASTER_KEY \
  -p 4000:4000 \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000 --num_workers 8

Using LiteLLM with the OpenAI SDK

No changes to your application code except the base_url:

from openai import OpenAI

# Before (direct to OpenAI):
# client = OpenAI(api_key="sk-...")

# After (through LiteLLM gateway):
client = OpenAI(
    api_key="your-litellm-virtual-key",  # Virtual key, not provider key
    base_url="http://localhost:4000",    # Your gateway URL
)

# All your existing code works unchanged
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this document..."}],
)

LiteLLM handles the provider selection, fallback, retries, and logging transparently.

Virtual Keys and Per-User Budget

LiteLLM supports "virtual keys" - keys issued to teams or users that map to real provider keys inside the gateway. This allows per-key budget limits without exposing actual provider keys to application teams.

import httpx

GATEWAY_URL = "http://localhost:4000"
MASTER_KEY = "your-master-key"


async def create_virtual_key(
    key_alias: str,
    max_budget_usd: float,
    team_id: str,
    models: list[str] = None,
    duration: str = "30d",  # key expires after 30 days
) -> dict:
    """Create a virtual key with a budget cap."""
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{GATEWAY_URL}/key/generate",
            headers={"Authorization": f"Bearer {MASTER_KEY}"},
            json={
                "key_alias": key_alias,
                "max_budget": max_budget_usd,
                "team_id": team_id,
                "models": models or ["gpt-4o", "gpt-4o-mini"],
                "duration": duration,
            },
        )
        return response.json()


async def get_key_spend(virtual_key: str) -> dict:
    """Get current spend for a virtual key."""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"{GATEWAY_URL}/key/info",
            headers={"Authorization": f"Bearer {MASTER_KEY}"},
            params={"key": virtual_key},
        )
        return response.json()

Model Routing Patterns

Not all routing decisions should be static. A sophisticated gateway routes based on request characteristics.

Pattern 1: Static Routing (simplest)

Route by explicit model name. Applications request gpt-4o or gpt-4o-mini directly. The gateway handles fallbacks if the primary is down.

When to use: applications know their quality/cost tradeoff requirements.

Pattern 2: Cost-Based Routing

Route cheap requests to cheap models, expensive requests to expensive models. Use a simple heuristic: input token count as a proxy for task complexity.

def route_by_cost(messages: list[dict], budget_tier: str) -> str:
    """Route to model based on budget tier and request size."""
    total_chars = sum(len(m.get("content", "")) for m in messages)
    estimated_tokens = total_chars // 4

    if budget_tier == "free":
        return "gpt-4o-mini"

    if budget_tier == "paid" and estimated_tokens < 2000:
        return "gpt-4o-mini"   # short paid requests -> cheap model

    return "gpt-4o"            # long or high-tier -> expensive model

Pattern 3: Quality Classifier Routing

Use a fast, cheap classifier to determine request complexity, then route to the appropriate model tier.

from openai import OpenAI
from enum import Enum


class ComplexityTier(Enum):
    SIMPLE = "simple"      # factual lookup, short formatting
    MODERATE = "moderate"  # summarization, code generation
    COMPLEX = "complex"    # multi-step reasoning, analysis


MODEL_FOR_TIER = {
    ComplexityTier.SIMPLE: "gpt-4o-mini",
    ComplexityTier.MODERATE: "gpt-4o-mini",
    ComplexityTier.COMPLEX: "gpt-4o",
}

CLASSIFIER_PROMPT = """Classify this user request into one complexity tier:
- SIMPLE: factual lookup, yes/no, short formatting, data extraction
- MODERATE: summarization, translation, code generation under 50 lines
- COMPLEX: multi-step reasoning, analysis, research, code architecture

User request: {query}

Respond with only one word: SIMPLE, MODERATE, or COMPLEX."""


class ComplexityRouter:
    def __init__(self, client: OpenAI):
        self.client = client
        self._cache: dict[str, ComplexityTier] = {}

    def classify(self, user_message: str) -> ComplexityTier:
        cache_key = user_message[:200]
        if cache_key in self._cache:
            return self._cache[cache_key]

        response = self.client.chat.completions.create(
            model="gpt-4o-mini",  # Always use cheap model for classification
            max_tokens=10,
            temperature=0,
            messages=[
                {
                    "role": "user",
                    "content": CLASSIFIER_PROMPT.format(query=user_message[:500]),
                }
            ],
        )

        tier_str = response.choices[0].message.content.strip().upper()
        try:
            tier = ComplexityTier[tier_str]
        except KeyError:
            tier = ComplexityTier.MODERATE  # default on parse failure

        self._cache[cache_key] = tier
        return tier

    def get_model(self, user_message: str) -> str:
        tier = self.classify(user_message)
        return MODEL_FOR_TIER[tier]

Pattern 4: A/B Routing

Route a percentage of traffic to a new model to compare quality before full rollout.

import hashlib
from dataclasses import dataclass
from typing import Optional


@dataclass
class ABExperiment:
    name: str
    control_model: str
    treatment_model: str
    treatment_fraction: float  # 0.0 to 1.0


class ABRouter:
    def __init__(self, experiments: list[ABExperiment]):
        self.experiments = {e.name: e for e in experiments}

    def route(
        self,
        experiment_name: str,
        user_id: str,
    ) -> tuple[str, str]:
        """
        Returns (model_name, variant) where variant is 'control' or 'treatment'.
        Uses user_id for stable assignment.
        """
        exp = self.experiments.get(experiment_name)
        if not exp:
            return "gpt-4o", "control"

        hash_val = int(
            hashlib.md5(f"{experiment_name}:{user_id}".encode()).hexdigest(), 16
        )
        fraction = (hash_val % 10000) / 10000

        if fraction < exp.treatment_fraction:
            return exp.treatment_model, "treatment"
        return exp.control_model, "control"

Circuit Breaker: Detecting Provider Outages

A circuit breaker monitors failure rates and automatically stops sending traffic to a degraded provider, giving it time to recover.

import time
import threading
from collections import deque
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    """
    Thread-safe circuit breaker for LLM provider calls.
    """

    def __init__(
        self,
        provider: str,
        failure_threshold: int = 5,
        window_seconds: int = 60,
        reset_timeout_seconds: int = 30,
        success_threshold: int = 2,
    ):
        self.provider = provider
        self.failure_threshold = failure_threshold
        self.window_seconds = window_seconds
        self.reset_timeout_seconds = reset_timeout_seconds
        self.success_threshold = success_threshold

        self.state = CircuitState.CLOSED
        self.failure_times: deque = deque()
        self.consecutive_successes = 0
        self.opened_at: Optional[float] = None
        self._lock = threading.Lock()

    def record_success(self):
        with self._lock:
            if self.state == CircuitState.HALF_OPEN:
                self.consecutive_successes += 1
                if self.consecutive_successes >= self.success_threshold:
                    self.state = CircuitState.CLOSED
                    self.failure_times.clear()
                    self.consecutive_successes = 0

    def record_failure(self):
        with self._lock:
            now = time.time()
            self.failure_times.append(now)

            cutoff = now - self.window_seconds
            while self.failure_times and self.failure_times[0] < cutoff:
                self.failure_times.popleft()

            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.OPEN
                self.opened_at = now
                self.consecutive_successes = 0
            elif self.state == CircuitState.CLOSED:
                if len(self.failure_times) >= self.failure_threshold:
                    self.state = CircuitState.OPEN
                    self.opened_at = now

    def allow_request(self) -> bool:
        with self._lock:
            if self.state == CircuitState.CLOSED:
                return True

            if self.state == CircuitState.OPEN:
                if time.time() - self.opened_at >= self.reset_timeout_seconds:
                    self.state = CircuitState.HALF_OPEN
                    self.consecutive_successes = 0
                    return True  # allow one probe request
                return False

            return True  # HALF_OPEN: allow limited requests

    @property
    def is_open(self) -> bool:
        return not self.allow_request()

Budget Management

Unlimited LLM spend is an engineering incident waiting to happen. A cost spike from a prompt injection attack, an infinite loop, or a sudden traffic surge can generate thousands of dollars in API charges before anyone notices.

import time
import redis.asyncio as aioredis


class BudgetEnforcer:
    """
    Per-key, per-project budget enforcement using Redis counters.
    """

    COST_PER_1K_INPUT = {
        "gpt-4o": 0.0025,
        "gpt-4o-mini": 0.00015,
        "claude-3-5-sonnet-20241022": 0.003,
        "claude-3-haiku-20240307": 0.00025,
    }
    COST_PER_1K_OUTPUT = {
        "gpt-4o": 0.010,
        "gpt-4o-mini": 0.00060,
        "claude-3-5-sonnet-20241022": 0.015,
        "claude-3-haiku-20240307": 0.00125,
    }

    def __init__(self, redis_client, alert_callback=None):
        self.redis = redis_client
        self.alert_callback = alert_callback

    def estimate_cost(
        self, model: str, input_tokens: int, output_tokens: int
    ) -> float:
        input_rate = self.COST_PER_1K_INPUT.get(model, 0.01)
        output_rate = self.COST_PER_1K_OUTPUT.get(model, 0.03)
        return input_rate * input_tokens / 1000 + output_rate * output_tokens / 1000

    async def check_budget(
        self,
        key_id: str,
        budget_limit_usd: float,
        estimated_cost: float,
    ) -> tuple[bool, float]:
        """Returns (allowed, current_spend)."""
        redis_key = f"budget:{key_id}:month"
        current_spend = float(await self.redis.get(redis_key) or 0)

        if current_spend + estimated_cost > budget_limit_usd:
            return False, current_spend

        # Alert at 80% consumed
        if current_spend > budget_limit_usd * 0.8 and self.alert_callback:
            await self.alert_callback(
                key_id=key_id, spent=current_spend, limit=budget_limit_usd
            )

        return True, current_spend

    async def record_spend(
        self,
        key_id: str,
        actual_cost: float,
        model: str,
        input_tokens: int,
        output_tokens: int,
    ):
        redis_key = f"budget:{key_id}:month"
        pipe = self.redis.pipeline()
        pipe.incrbyfloat(redis_key, actual_cost)
        pipe.expire(redis_key, 30 * 24 * 3600)
        pipe.hincrby(f"budget:stats:{key_id}", f"tokens_input:{model}", input_tokens)
        pipe.hincrby(f"budget:stats:{key_id}", f"tokens_output:{model}", output_tokens)
        await pipe.execute()

SLA Monitoring Per Provider

Track per-provider reliability to make routing decisions based on real performance data.

import statistics
import time
from dataclasses import dataclass, field


class ProviderSLA:
    def __init__(self, provider: str, window_seconds: int = 300):
        self.provider = provider
        self.window_seconds = window_seconds
        self.latency_samples: list[float] = []
        self.error_count = 0
        self.total_requests = 0
        self.window_start = time.time()

    def record(self, latency_ms: float, success: bool):
        now = time.time()
        if now - self.window_start > self.window_seconds:
            self.latency_samples = []
            self.error_count = 0
            self.total_requests = 0
            self.window_start = now

        self.total_requests += 1
        if success:
            self.latency_samples.append(latency_ms)
        else:
            self.error_count += 1

    @property
    def error_rate(self) -> float:
        if self.total_requests == 0:
            return 0.0
        return self.error_count / self.total_requests

    @property
    def p50_latency_ms(self) -> float:
        if not self.latency_samples:
            return 0.0
        return statistics.median(self.latency_samples)

    @property
    def p95_latency_ms(self) -> float:
        if len(self.latency_samples) < 20:
            return 0.0
        return sorted(self.latency_samples)[int(len(self.latency_samples) * 0.95)]

    def as_dict(self) -> dict:
        return {
            "provider": self.provider,
            "error_rate": round(self.error_rate, 4),
            "p50_ms": round(self.p50_latency_ms, 1),
            "p95_ms": round(self.p95_latency_ms, 1),
            "total_requests": self.total_requests,
        }

Putting It Together: Full Gateway Client

import asyncio
import time
import hashlib
from typing import Optional
from openai import AsyncOpenAI
import structlog

log = structlog.get_logger()


class LLMGatewayClient:
    """
    Production LLM gateway client with routing, circuit breaking, and budget enforcement.
    Wraps LiteLLM proxy with additional application-level logic.
    """

    def __init__(
        self,
        gateway_url: str,
        virtual_key: str,
        budget_enforcer: BudgetEnforcer,
        router: ComplexityRouter,
        circuit_breakers: dict[str, CircuitBreaker],
        sla_trackers: dict[str, ProviderSLA],
    ):
        self.client = AsyncOpenAI(api_key=virtual_key, base_url=gateway_url)
        self.budget = budget_enforcer
        self.router = router
        self.circuit_breakers = circuit_breakers
        self.sla_trackers = sla_trackers

    async def complete(
        self,
        messages: list[dict],
        key_id: str,
        budget_limit_usd: float = 10.0,
        force_model: Optional[str] = None,
        max_tokens: int = 1024,
        temperature: float = 0.7,
    ) -> dict:
        user_message = next(
            (m["content"] for m in messages if m["role"] == "user"), ""
        )
        model = force_model or self.router.get_model(user_message)

        # Budget pre-flight
        estimated_cost = self.budget.estimate_cost(
            model, len(user_message) // 4, max_tokens
        )
        allowed, current_spend = await self.budget.check_budget(
            key_id, budget_limit_usd, estimated_cost
        )
        if not allowed:
            raise RuntimeError(
                f"Budget exceeded: ${current_spend:.2f} / ${budget_limit_usd:.2f}"
            )

        # Circuit breaker check
        provider = "openai" if "gpt" in model else "anthropic"
        cb = self.circuit_breakers.get(provider)
        if cb and cb.is_open:
            fallback = (
                "claude-3-5-sonnet-20241022" if "gpt" in model else "gpt-4o"
            )
            log.warning(
                "circuit_open_fallback",
                provider=provider,
                original=model,
                fallback=fallback,
            )
            model = fallback

        start = time.monotonic()
        try:
            response = await self.client.chat.completions.create(
                model=model,
                messages=messages,
                max_tokens=max_tokens,
                temperature=temperature,
            )
            latency_ms = (time.monotonic() - start) * 1000

            if cb:
                cb.record_success()
            if provider in self.sla_trackers:
                self.sla_trackers[provider].record(latency_ms, True)

            usage = response.usage
            actual_cost = self.budget.estimate_cost(
                model, usage.prompt_tokens, usage.completion_tokens
            )
            await self.budget.record_spend(
                key_id, actual_cost, model,
                usage.prompt_tokens, usage.completion_tokens
            )

            log.info(
                "llm_request_success",
                model=model,
                latency_ms=round(latency_ms, 1),
                input_tokens=usage.prompt_tokens,
                output_tokens=usage.completion_tokens,
                cost_usd=round(actual_cost, 6),
            )

            return {
                "content": response.choices[0].message.content,
                "model": model,
                "usage": {
                    "input_tokens": usage.prompt_tokens,
                    "output_tokens": usage.completion_tokens,
                },
                "cost_usd": actual_cost,
                "latency_ms": latency_ms,
            }

        except Exception as e:
            latency_ms = (time.monotonic() - start) * 1000
            if cb:
                cb.record_failure()
            if provider in self.sla_trackers:
                self.sla_trackers[provider].record(latency_ms, False)
            log.error("llm_request_failed", model=model, error=str(e))
            raise

Production Configuration Checklist

Common Mistakes

:::danger Putting provider API keys directly in application services Every service that holds a raw provider API key is a liability. If any service is compromised, the key is exposed. A gateway with virtual keys means a compromised application service exposes only a scoped virtual key with budget limits, not an unbounded raw provider key. :::

:::danger No budget hard limit - only soft alerts An alert at 80% of budget is useful. But if a runaway process generates 1,000 requests per minute during an incident, it can blow through 20% of your budget in minutes before anyone responds to the alert. Implement a hard block at 100% of budget. Accept that a small number of requests will fail - that is preferable to a $5,000 unexpected charge. :::

:::warning Routing all traffic to the cheap model and expecting quality parity GPT-4o-mini and Claude 3 Haiku are excellent for classification, extraction, and short Q&A. They underperform GPT-4o and Claude 3.5 Sonnet on multi-step reasoning, long-context analysis, and complex code generation. A static "always use cheap model" policy will produce quality regressions on tasks that warrant the expensive model. :::

:::warning Circuit breaker thresholds that are too sensitive A circuit breaker that opens on 2 failures in 60 seconds will trigger on normal transient errors. Set failure thresholds that distinguish genuine provider degradation from noise: 5 failures in 60 seconds is a reasonable starting point. Test thresholds in load testing before relying on them in production. :::

:::danger Testing fallback only in your head Fallback chains feel reassuring on the architecture diagram. They often fail silently in production when the fallback model has a different parameter naming convention, a different context window, or a rate limit that is also exhausted during the same incident. Test failover by deliberately taking down your primary in staging. Verify the fallback produces acceptable outputs, not just that the API call succeeds. :::

Interview Questions

Q: Why would you build an LLM gateway instead of just adding retry logic to each service?

A: Retry logic per service solves one narrow problem. A gateway solves a class of problems. Per-service retry still leaves you with: multiple places to manage API keys (a security problem); no unified cost tracking (you do not know which feature is driving spend); no cross-service rate limiting (service A can exhaust your quota while service B is rate-limited); no ability to A/B test models across features; and no circuit breaking that responds to provider-level incidents rather than per-request failures. A gateway centralizes these cross-cutting concerns in one place. The operational cost of maintaining a gateway is paid once. The operational cost of implementing these concerns per-service is paid N times and produces N independently broken implementations.

---

Q: How does a circuit breaker for LLM providers differ from a standard HTTP circuit breaker?

A: A standard HTTP circuit breaker trips on connection failures and 5xx responses. LLM providers add nuance. A 429 (rate limited) is a temporary condition that warrants backoff, not circuit tripping - you should queue and retry, not switch to a different provider. A 500 with "model overloaded" is different from a 500 from a persistent bug. A 200 response with "I cannot help with that due to policy" is a successful HTTP call but a semantic failure. LLM circuit breakers need to distinguish: rate limits (backoff and retry), capacity errors (circuit trip and failover), policy refusals (application-level handling, not infrastructure), and timeouts (the most common cause of circuit trips because inference time is unpredictable). Additionally, the half-open probe for LLM providers should use a cheap model or a simple test prompt, not the full production request, to minimize cost during recovery probing.

---

Q: You are running an LLM gateway and your primary provider goes down. Walk through what happens in your system.

A: Step one: the first few requests to the primary provider return 5xx errors or time out. Each failure increments the failure counter in the circuit breaker. Step two: after the failure threshold is crossed, the circuit opens. All subsequent requests fail fast without attempting the primary - detected in microseconds instead of waiting for a 30-second timeout. Step three: the routing layer sees the circuit is open and routes to the fallback provider. Step four: requests continue to flow, now routed to the secondary. Latency and cost may differ - log these differences. Step five: after the reset timeout, the circuit transitions to half-open. One probe request goes to the primary. If it succeeds, the circuit closes and traffic returns. If it fails, the circuit re-opens. Throughout this process, an alert fires on the circuit state change so the on-call engineer is aware, even though users are not impacted.

---

Q: How would you design budget management for a multi-tenant SaaS product where each customer has a different LLM usage tier?

A: Three-tier budget architecture. First, the global budget: a hard limit on total monthly spend across all customers - protects against exploits or infrastructure failures generating runaway traffic. Second, per-customer budgets: each account has a monthly LLM budget based on their subscription tier (free: $2/month; starter:$20/month; professional: $200/month). Enforced at the gateway via virtual keys. When a customer's key hits 80% of budget, send an alert. At 100%, fail requests with a clear error message and upgrade prompt. Third, per-feature sub-limits within a customer account to prevent one feature from consuming the entire monthly budget in one batch operation. Implementation: Redis counters keyed by budget:{customer_id}:{month}. Increment on every request completion, check before every request. Use Lua scripts in Redis for atomic check-and-increment to avoid race conditions.

---

Q: What is model routing by complexity classifier, and what are its limitations?

A: A complexity classifier is a small, fast LLM (or traditional ML classifier) that reads the user request and assigns it to a tier: simple, moderate, or complex - then routes to the corresponding model. The benefit is automatic cost optimization without requiring applications to specify which model to use. The limitations are significant. First, the classifier can be wrong - a question that looks simple may require careful output formatting, while a complex-looking question may be answerable by a cheap model. Second, the classifier adds latency - for interactive applications where total latency is 1–3 seconds, adding 150ms for classification is a 5–15% overhead. Third, the classifier is itself a model that needs evaluation and maintenance. Fourth, complexity is context-dependent: whether a question is complex depends on what the system prompt and conversation history say. The classifier needs the full context to make a good prediction, which increases its own token cost.

---

Q: How do you handle the case where all your LLM providers are degraded simultaneously?

A: This happens more often than people expect - major incidents at OpenAI and Anthropic have overlapped in the past. Defensive strategies: First, maintain a self-hosted model as last resort. A locally-hosted Llama 3.1 70B on GPU instances is weaker than GPT-4o but infinitely better than 100% errors. Pre-provision the infrastructure so it is ready to receive traffic. Second, implement graceful degradation at the application level: for non-critical features, return a cached response or a "service temporarily unavailable" message instead of failing the entire request. Reserve remaining LLM capacity for critical features. Third, implement a queue with backpressure: during degradation, instead of returning errors immediately, queue requests and return an estimated wait time. Many users will wait for a short outage. Fourth, communicate proactively - status page update, in-app banner, email to affected enterprise customers. The cost of one proactive communication is much lower than reactive support tickets.

:::tip 🎮 Interactive Playground

Visualize this concept: Try the Prompt Routing demo on the EngineersOfAI Playground - no code required.

:::