Latency and Cost Tradeoffs

The Invoice That Changed Everything

At 11 PM on a Thursday, the head of engineering receives an automated alert: projected monthly spend on LLM APIs has crossed $50,000 - up from$8,000 the previous month. The product launched 6 weeks ago and growth has been good, but nothing that explains a 6x cost increase. Something is structurally wrong.

The engineering team starts digging. They pull up token usage metrics and find the culprit immediately: average input tokens per request have grown from 800 to 4,200. Users are having longer conversations, and the team is sending the full conversation history on every turn. A user who has been chatting for 30 minutes now has 15 turns of history - roughly 3,500 tokens - that gets sent to the model on every single new message. The 16th message costs 16x what the first message cost.

But the cost problem is just the beginning. They also discover that P99 latency has climbed to 28 seconds. Customer support tickets about "the AI being slow" are up 400%. The latency and the cost are the same bug: unbounded context growth. Fixing the context management will fix both.

This story repeats across teams building LLM products. The economics of LLM APIs are unlike any prior API the industry has dealt with. The cost of a single request can vary by 100x depending on how you use it. A GET request to a database is essentially free and deterministic. An LLM request can cost anywhere from $0.001 to$1.00, and the difference lies entirely in how you construct the call.

Understanding the full cost and latency breakdown - and knowing which lever to pull first - is one of the most practically valuable skills for an LLM engineer.

Why This Exists

Traditional web services have predictable cost models: compute scales with requests per second, memory scales with data held in RAM, storage costs are flat. You can estimate your infrastructure bill within 20% before launch.

LLM APIs break this model. The cost per request is a function of the request content - specifically, the number of tokens. And tokens are hard to predict. A user with a verbose style generates 3x more tokens than a concise user asking the same question. A system prompt that grows over time as you add few-shot examples can double your costs without any change in traffic.

Worse, LLM API cost structures penalize the most natural thing you want to do: give the model more context. More context = better quality, but also more cost and more latency. The three objectives - quality, latency, cost - are in tension with each other, and navigating that tension is the core skill this lesson teaches.

The Three-Axis Tradeoff

Every LLM system design decision sits on a triangle with three vertices:

          QUALITY
             /\
            /  \
           /    \
          /  pick 2  \
         /        \
        /____________\
    LATENCY        COST

You cannot optimize all three simultaneously. In practice, you choose which two matter most for your use case:

Priority	Sacrifice	Example Product
Quality + Latency	Cost	Real-time coding assistant (developer pays premium)
Quality + Cost	Latency	Overnight document analysis batch job
Latency + Cost	Quality	Autocomplete suggestions (speed matters, good enough is enough)

The most common mistake is not making this choice explicitly. When you optimize for all three, you get an incoherent system that is mediocre on all axes.

Latency Decomposition

The total latency of an LLM API call is the sum of five distinct components. Each has different drivers and different optimization strategies.

$L_{total} = L_{network} + L_{prefill} + L_{decode} + L_{post}$

where:

• $L_{network}$ = round-trip network time to the LLM API
• $L_{prefill}$ = time to process the input prompt (scales with input tokens)
• $L_{decode}$ = time to generate output tokens (scales with output length)
• $L_{post}$ = post-processing: parsing, validation, formatting

Network RTT: 20–100ms

The round-trip time from your servers to the LLM API provider's data center. For US-East to OpenAI: typically 20-40ms. For EU to OpenAI US: 80-120ms.

Optimization: Deploy your LLM service in the same region as the provider. Use providers with regional endpoints (Anthropic EU, Azure OpenAI regional deployments).

Prefill Time: 50–500ms (scales with input tokens)

The time for the model to process the entire input prompt and compute the KV cache for all input tokens. This is parallelizable on the GPU, so it scales roughly as $O(n^{1.1})$ with input length - nearly linear but with some quadratic attention overhead for very long prompts.

Key insight: A 4,000-token prompt takes roughly 4x longer to prefill than a 1,000-token prompt. Reducing prompt length reduces both prefill latency and input token cost simultaneously.

Optimization: Prompt compression, history truncation, provider-level prompt caching (if the prefix is stable across requests, the KV cache is reused and prefill is essentially free).

Time to First Token (TTFT)

$TTFT = L_{network} + L_{prefill}$

This is the user-perceived "response start time." For streaming interfaces, this is the critical metric - users see the first token appear and feel that the system is working. TTFT of under 1 second feels instant. TTFT of 3+ seconds feels broken.

Target: TTFT under 1 second for interactive applications.

Decode Time: 100ms–30s (scales with output length)

The time to generate all output tokens. Each token is generated sequentially (autoregressive decoding), so decode time scales linearly with output length. The per-token cost is called TPOT (Time Per Output Token).

$L_{decode} = N_{output} \times TPOT$

For GPT-4o, TPOT is typically 15-30ms per token. For a 500-token response:

• TPOT = 20ms → decode time = 10 seconds
• TPOT = 30ms → decode time = 15 seconds

Key insight: Output tokens are both slower AND more expensive than input tokens. Controlling output length is the single highest-impact optimization for both latency and cost.

Post-Processing: 5–50ms

Parsing JSON, validating schemas, extracting fields, routing the output. Usually negligible unless your parsing involves LLM calls (e.g., using GPT to validate another GPT's output - avoid this).

Typical total latency by scenario:

Scenario	Input Tokens	Output Tokens	Typical Total Latency
Short classification	500	10	500ms–1s
Chat response	2,000	200	3–8s
Document summary	8,000	500	8–20s
Long-form generation	4,000	2,000	25–60s

Cost Decomposition

LLM API pricing (as of GPT-4o class models, 2024-2025) follows this general structure:

Token Type	Approximate Range	Notes
Input tokens	$1–5 / million	What the model reads
Output tokens	$5–15 / million	What the model generates
Cached input tokens	$0.10–0.50 / million	90% discount on cached prefix
Embeddings	$0.02–0.13 / million	Much cheaper
Image tokens	Varies	Per image, or per tile

The fundamental asymmetry: Output tokens cost 3-5x more than input tokens, per token. This means generating 1,000 output tokens costs roughly as much as processing 4,000-5,000 input tokens. Every sentence the model writes is expensive.

Estimating Costs

# cost_estimator.py

PRICING = {
    "gpt-4o": {
        "input": 2.50 / 1_000_000,       # $2.50 per million input tokens
        "output": 10.00 / 1_000_000,      # $10.00 per million output tokens
        "cached_input": 1.25 / 1_000_000, # 50% discount with prompt caching
    },
    "gpt-4o-mini": {
        "input": 0.15 / 1_000_000,
        "output": 0.60 / 1_000_000,
        "cached_input": 0.075 / 1_000_000,
    },
    "claude-3-5-sonnet": {
        "input": 3.00 / 1_000_000,
        "output": 15.00 / 1_000_000,
        "cached_input": 0.30 / 1_000_000,  # 90% discount with Anthropic caching
    },
    "claude-3-haiku": {
        "input": 0.25 / 1_000_000,
        "output": 1.25 / 1_000_000,
        "cached_input": 0.03 / 1_000_000,
    },
}


def estimate_cost(
    model: str,
    input_tokens: int,
    output_tokens: int,
    cached_input_tokens: int = 0,
) -> dict:
    pricing = PRICING[model]
    non_cached_input = input_tokens - cached_input_tokens

    input_cost = non_cached_input * pricing["input"]
    cached_cost = cached_input_tokens * pricing["cached_input"]
    output_cost = output_tokens * pricing["output"]
    total = input_cost + cached_cost + output_cost

    return {
        "model": model,
        "input_cost": input_cost,
        "cached_input_cost": cached_cost,
        "output_cost": output_cost,
        "total_cost": total,
        "cost_per_1k_total_tokens": total / (input_tokens + output_tokens) * 1000,
    }


def monthly_cost_projection(
    requests_per_day: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    model: str,
    cache_hit_rate: float = 0.0,
    cached_prefix_tokens: int = 0,
) -> dict:
    """Project monthly cost given traffic and token patterns."""
    single_request = estimate_cost(
        model=model,
        input_tokens=avg_input_tokens,
        output_tokens=avg_output_tokens,
        cached_input_tokens=int(cached_prefix_tokens * cache_hit_rate),
    )

    daily_cost = requests_per_day * single_request["total_cost"]
    monthly_cost = daily_cost * 30

    return {
        "requests_per_day": requests_per_day,
        "avg_input_tokens": avg_input_tokens,
        "avg_output_tokens": avg_output_tokens,
        "cost_per_request": single_request["total_cost"],
        "daily_cost": daily_cost,
        "monthly_cost": monthly_cost,
        "monthly_input_tokens": requests_per_day * 30 * avg_input_tokens,
        "monthly_output_tokens": requests_per_day * 30 * avg_output_tokens,
    }


# Example: chat app with 10,000 daily active users, 5 messages per user per day
result = monthly_cost_projection(
    requests_per_day=50_000,
    avg_input_tokens=3_000,
    avg_output_tokens=500,
    model="gpt-4o",
    cache_hit_rate=0.7,
    cached_prefix_tokens=800,  # system prompt length
)
# ~$1,950/month with caching, vs ~$3,750/month without caching

Latency Optimization Strategies (Ranked by Impact)

1. Reduce Output Length (Highest Impact)

The single most effective latency and cost optimization. Every token you do not generate saves both time and money.

Techniques:

• Set an appropriate max_tokens limit for each call type
• Use structured output (JSON) instead of prose explanations
• Add instructions to the system prompt: "Be concise. Respond in under 100 words."
• Use the model as a classifier (1-3 token output) instead of a generator where possible

# Instead of asking for prose explanation, use structured output:
from pydantic import BaseModel
from openai import AsyncOpenAI

class CodeReviewResult(BaseModel):
    has_bug: bool
    bug_type: str | None      # "null_pointer" | "off_by_one" | "race_condition"
    severity: str | None      # "low" | "medium" | "high"
    fix_suggestion: str | None  # Max 50 words

client = AsyncOpenAI()
result = await client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Analyze the code. Be concise."},
        {"role": "user", "content": f"Review this:\n{code}"},
    ],
    response_format=CodeReviewResult,
    max_tokens=150,  # Enforce output budget
)

2. Reduce Prompt Length

Shorter prompts = faster prefill + lower input cost. A prompt that is 2x shorter is 2x cheaper and ~1.8x faster to prefill.

Techniques:

• Implement conversation history truncation (see Module 03)
• Compress few-shot examples: one well-chosen example beats three mediocre ones
• Remove redundant instructions (system prompts accumulate cruft over time)
• Use LLMLingua for automatic prompt compression (3-20x compression with ~5% quality loss)

# Simple prompt length analyzer
import tiktoken

def analyze_prompt_tokens(messages: list[dict], model: str = "gpt-4o") -> dict:
    enc = tiktoken.encoding_for_model(model)

    breakdown = {}
    total = 0
    for msg in messages:
        tokens = len(enc.encode(msg["content"]))
        role = msg["role"]
        breakdown[role] = breakdown.get(role, 0) + tokens
        total += tokens

    return {
        "total_tokens": total,
        "by_role": breakdown,
        "estimated_prefill_ms": total * 0.08,  # rough: 0.08ms per token
        "estimated_input_cost_gpt4o": total * 2.50 / 1_000_000,
    }

3. Use a Faster/Smaller Model for Simpler Subtasks

Not every step in your pipeline needs GPT-4. Model tiering is one of the highest-ROI optimizations.

Task Type	Recommended Model	Reason
Intent classification	GPT-4o-mini / Claude Haiku	Simple pattern matching
Extracting structured fields	GPT-4o-mini	Well-defined task
Generating first-pass draft	GPT-4o-mini	Cheap, fast, good enough
Refining quality-sensitive output	GPT-4o / Claude Sonnet	Worth the cost
Novel reasoning, complex analysis	GPT-4o / Claude Sonnet	Required

The practical question is: "What is the cheapest model that passes my quality bar on my evaluation set?" Run an offline evaluation on your golden dataset and measure quality degradation per model. The quality-cost frontier is often surprising - small models on well-constrained tasks perform remarkably well.

4. Streaming (Reduces Perceived Latency)

Streaming does not reduce total latency. It reduces perceived latency by showing users the first tokens quickly, which transforms the user experience.

With streaming enabled:

• TTFT: 200-800ms (excellent UX)
• Total time: same as without streaming

Without streaming:

• TTFT: same as total time (5-20s for typical requests)
• Users stare at a spinner for the entire generation duration

Enable streaming for all interactive features. Disable it for batch pipelines where the consumer needs the full result before proceeding.

5. Prompt Caching

If your system prompt is long and stable, prompt caching is effectively free latency reduction and cost reduction simultaneously.

Anthropic: Minimum 1024 tokens of cached prefix. Cache TTL is 5 minutes (ephemeral) - renewed on each cache hit. 90% cost discount on cached tokens. Cache hit is detectable in the response cache_read_input_tokens field.

OpenAI: Minimum 1024 tokens. 50% discount on cached tokens. Cache is automatic (no explicit opt-in).

Design your prompts for cache hit rate:

1. Put all stable content at the beginning: system instructions, persona, few-shot examples, tool definitions
2. Put all dynamic content at the end: retrieved context, conversation history, current user message
3. Minimize changes to the stable prefix - even a single character change invalidates the cache

6. Parallel LLM Calls

If your workflow has independent subtasks, fire them concurrently.

import asyncio
from openai import AsyncOpenAI

async def analyze_document_parallel(document: str) -> dict:
    """Run multiple analyses concurrently instead of sequentially."""
    client = AsyncOpenAI()

    async def summarize():
        r = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Summarize in 3 bullet points."},
                {"role": "user", "content": document},
            ],
            max_tokens=200,
        )
        return r.choices[0].message.content

    async def extract_entities():
        r = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Extract: people, orgs, dates. JSON only."},
                {"role": "user", "content": document},
            ],
            max_tokens=300,
        )
        return r.choices[0].message.content

    async def classify_sentiment():
        r = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Classify: positive/neutral/negative. One word."},
                {"role": "user", "content": document},
            ],
            max_tokens=5,
        )
        return r.choices[0].message.content

    # Run all three concurrently - total time is the slowest single call
    summary, entities, sentiment = await asyncio.gather(
        summarize(), extract_entities(), classify_sentiment()
    )

    return {"summary": summary, "entities": entities, "sentiment": sentiment}
    # Sequential: ~9s total. Parallel: ~3s total.

Cost Optimization Strategies

Model Tiering: The Quality-Cost Frontier

The goal is to route each request to the cheapest model that meets your quality bar. This requires building an offline evaluation first.

# quality_cost_frontier.py

async def find_optimal_model(
    eval_dataset: list[dict],
    quality_threshold: float = 0.85,
) -> str:
    """Find the cheapest model that meets the quality threshold."""
    models_to_evaluate = [
        "gpt-4o-mini",
        "gpt-4o",
        "claude-3-haiku-20240307",
        "claude-3-5-sonnet-20241022",
    ]

    results = {}
    for model in models_to_evaluate:
        scores = []
        costs = []

        for sample in eval_dataset:
            response = await call_model(model, sample["input"])
            score = sample["quality_fn"](response, sample["expected_output"])
            cost = estimate_cost(model, sample["input"], response)
            scores.append(score)
            costs.append(cost)

        avg_quality = sum(scores) / len(scores)
        avg_cost = sum(costs) / len(costs)
        results[model] = {"quality": avg_quality, "avg_cost": avg_cost}

    # Find cheapest model above threshold
    qualifying = {
        m: r for m, r in results.items() if r["quality"] >= quality_threshold
    }
    if not qualifying:
        return max(results, key=lambda m: results[m]["quality"])

    return min(qualifying, key=lambda m: qualifying[m]["avg_cost"])

Batch API (50% Cost Discount)

Both OpenAI and Anthropic offer batch APIs for non-real-time workloads at a 50% discount. Use this for:

• Overnight document processing
• Bulk data enrichment
• Evaluation runs
• Preprocessing pipelines

The tradeoff: 24-hour turnaround time. Not suitable for user-facing features.

Prompt Compression with LLMLingua

LLMLingua (Microsoft Research, 2023) uses a small LM to compress prompts by removing tokens that contribute little to the final answer. Achieves 3-20x compression with less than 5% quality loss on most tasks.

# pip install llmlingua
from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
    device_map="cpu",
)

def compress_prompt(
    system_prompt: str,
    context: str,
    question: str,
    target_compression_ratio: float = 0.5,
) -> dict:
    result = compressor.compress_prompt(
        context,
        instruction=system_prompt,
        question=question,
        target_token=int(len(context.split()) * target_compression_ratio),
        condition_in_question="after_condition",
    )

    original_tokens = result["origin_tokens"]
    compressed_tokens = result["compressed_tokens"]
    compression = 1 - (compressed_tokens / original_tokens)

    return {
        "compressed_prompt": result["compressed_prompt"],
        "original_tokens": original_tokens,
        "compressed_tokens": compressed_tokens,
        "compression_ratio": compression,
        "estimated_savings_usd": compression * original_tokens * 2.50 / 1_000_000,
    }

Production: SLOs, Cost Budgets, and Alerting

Defining SLOs for LLM Features

An SLO (Service Level Objective) for an LLM feature should cover three dimensions:

Dimension	Metric	Example Target
Latency	TTFT P95	under 1.5 seconds
Latency	Total response P95	under 8 seconds
Availability	Success rate	99.5%
Cost	Cost per request	under $0.05
Quality	User satisfaction	above 4.0/5.0

Cost Budgets Per Feature

Assign a cost budget to each product feature independently. This makes cost spikes attributable and controllable.

# cost_budget_enforcer.py
from datetime import date

FEATURE_BUDGETS = {
    "chat": {"daily_usd": 500, "per_request_usd": 0.10},
    "summarize": {"daily_usd": 200, "per_request_usd": 0.05},
    "code_review": {"daily_usd": 1000, "per_request_usd": 0.20},
    "rag_search": {"daily_usd": 300, "per_request_usd": 0.03},
}

async def check_budget(
    feature: str,
    estimated_cost: float,
    redis_client,
) -> bool:
    """Returns False if budget exceeded, True if within budget."""
    budget = FEATURE_BUDGETS.get(feature, {})

    # Check per-request budget
    if estimated_cost > budget.get("per_request_usd", float("inf")):
        return False

    # Check daily budget
    key = f"cost:{feature}:{date.today().isoformat()}"
    daily_spend = float(await redis_client.get(key) or 0)
    if daily_spend + estimated_cost > budget.get("daily_usd", float("inf")):
        return False

    # Increment counter
    await redis_client.incrbyfloat(key, estimated_cost)
    await redis_client.expire(key, 86400 * 2)  # 48h TTL
    return True

Latency Profiler

# latency_profiler.py
import time
from dataclasses import dataclass

@dataclass
class LatencyBreakdown:
    session_load_ms: float = 0
    rag_retrieval_ms: float = 0
    prompt_assembly_ms: float = 0
    llm_prefill_ms: float = 0   # approximated as TTFT
    llm_decode_ms: float = 0    # total - TTFT
    post_process_ms: float = 0
    total_ms: float = 0
    input_tokens: int = 0
    output_tokens: int = 0


async def profiled_llm_call(
    messages: list[dict], model: str
) -> tuple[str, LatencyBreakdown]:
    client = AsyncOpenAI()
    bd = LatencyBreakdown()
    wall_start = time.perf_counter()
    first_token_time = None
    chunks = []

    stream = await client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
        stream_options={"include_usage": True},
    )

    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
                bd.llm_prefill_ms = (first_token_time - wall_start) * 1000
            chunks.append(chunk.choices[0].delta.content)
        if chunk.usage:
            bd.input_tokens = chunk.usage.prompt_tokens
            bd.output_tokens = chunk.usage.completion_tokens

    bd.total_ms = (time.perf_counter() - wall_start) * 1000
    bd.llm_decode_ms = bd.total_ms - bd.llm_prefill_ms

    return "".join(chunks), bd

Common Mistakes

:::danger Mistake: Sending full conversation history on every request This is the most common cause of cost explosions. A 50-turn conversation with 200 tokens per turn is 10,000 tokens of history. At GPT-4o rates, that is $0.025 of input cost - per request. With 10,000 requests per day, that is$250/day just from history context that the model largely ignores. Implement truncation from day one. :::

:::danger Mistake: No max_tokens limit Without a max_tokens limit, the model will generate until it reaches the context window limit or produces an end-of-sequence token. A model that decides to be "thorough" can generate 4,000-token responses when you wanted 200 tokens. This is a 20x cost overrun on output tokens. Always set max_tokens appropriate to the task. :::

:::warning Mistake: Optimizing latency before measuring "We should use a smaller model to reduce latency" - maybe, but which component of latency is actually dominant? If your P99 latency is 20 seconds and 18 seconds is decode time from a 3,000-token output, switching to a faster model will help modestly. But adding max_tokens=300 can give you a 6x latency reduction from a single line change. Measure first, then optimize the dominant term. :::

:::warning Mistake: Using streaming for batch pipelines Streaming adds per-connection overhead and complicates error handling. For batch jobs where the consumer needs the complete response before proceeding - document classification, field extraction, data enrichment - use non-streaming calls. Streaming is only valuable when a human is watching the response in real-time. :::

Interview Q&A

Q1: A product team says their chat feature is "too slow and too expensive." Walk me through how you would diagnose and fix this.

First, I want to separate the latency problem from the cost problem - they often have the same root cause but the fixes are sequenced differently.

For cost: pull token usage metrics. Break down input tokens by source: system prompt, conversation history, RAG context, and current message. History is almost always the largest and fastest-growing component. If average input tokens per request are growing over time, you have an unbounded history accumulation problem. Fix: implement sliding window or summarization-based history truncation.

For latency: measure TTFT vs total response time separately. If TTFT is good (under 1s) but total time is high, the problem is output length - add max_tokens. If TTFT is high, the problem is input token count (long prompts, slow prefill) or provider-side queuing. If total time has high variance (P50 ok, P99 terrible), the issue is likely provider rate limiting or model inference instability.

Fix sequence: (1) output length limits, (2) history truncation, (3) enable streaming if not already, (4) prompt caching for system prompt, (5) model downgrade for simpler queries.

Q2: Explain the difference between TTFT and total response latency, and when each matters.

TTFT (Time to First Token) is the time from when the request is sent to when the first token is received. Total latency is the time until the last token is received.

TTFT matters for interactive UX. When a user hits send, they are waiting to see the system respond. A TTFT of under 1 second feels responsive - text starts appearing immediately. A TTFT of 3+ seconds feels broken, even if the total response takes 10 seconds.

Total latency matters when the consumer needs the full response before proceeding - parsing JSON, feeding into a downstream step, generating a report. Also relevant for SLO compliance for non-interactive tasks.

In practice, enabling streaming transforms the user experience primarily by improving perceived TTFT, not actual total latency. The model still takes the same time to generate everything - but the user sees progress from the start.

Q3: If you had to cut LLM API costs by 50% without reducing quality, what would your strategy be?

I would work through this in priority order:

First, add prompt caching. If the system prompt is over 1,000 tokens and the same users make multiple requests per session, prompt caching alone can yield 30-60% cost reduction on input tokens. Cost: zero engineering effort, just ensure stable prefix design.

Second, implement history truncation if not already in place. Truncating to last 10 turns typically reduces input tokens by 40-60% for active users with minimal quality impact.

Third, evaluate model downgrade for specific features. Run offline evaluation comparing GPT-4o vs GPT-4o-mini on a golden dataset. For classification, extraction, and simple Q&A, mini-class models typically match full-class models at 1/10th the cost.

Fourth, add semantic caching for repeated queries. In most products, 20-40% of queries are semantically similar to previous queries. A semantic cache with 0.92 cosine similarity threshold avoids LLM calls entirely for these.

Combined, these four measures typically achieve 50-70% cost reduction without measurable quality degradation.

Q4: What is LLMLingua and when would you use it?

LLMLingua (Microsoft Research, 2023) uses a small language model to score the importance of each token in the prompt and removes low-importance tokens while preserving the information needed for the downstream task. It achieves 3-20x compression with less than 5% quality loss on most benchmarks.

I would use it in two scenarios: (1) when RAG retrieval returns very long document chunks that need injection into the prompt - LLMLingua can compress retrieved context by 5-10x, dramatically reducing both cost and prefill time; (2) when dealing with legacy systems where conversation history cannot be truncated more aggressively without breaking coherence.

I would NOT use it for system prompts (better to rewrite them to be concise) or for short prompts under 500 tokens (compression overhead not worth it).

Q5: How would you set up cost alerting for an LLM product to prevent bill shock?

Three layers of alerting:

Daily budget alerts: Track per-feature daily spend in Redis. Alert via Slack at 70% of daily budget, hard-stop new requests at 95%. This catches runaway costs before they compound.

Cost anomaly detection: Compare daily cost per request to the 7-day rolling average. Alert if cost per request increases by more than 30% - this signals prompt changes that accidentally increased token usage.

Token usage trend alerts: Track average input token count per request over time. If this is increasing week-over-week, you have a context accumulation problem that will compound. Alert before it becomes a crisis.

Additionally, set cloud provider budget alerts at a hard dollar cap as a last-resort safety net. Never rely solely on application-level monitoring - always have cloud billing alerts as a backstop.

:::tip 🎮 Interactive Playground

Visualize this concept: Try the Inference Batching & Throughput demo on the EngineersOfAI Playground - no code required.

:::