:::tip 🎮 Interactive Playground Visualize this concept: Try the Inference Batching demo on the EngineersOfAI Playground - no code required. :::

Batching Strategies for Inference

The GPU Sitting Idle While You Pay for It

The alert comes in at 9 AM Monday: inference costs have doubled compared to last quarter, but traffic has only grown 30%. The ML platform team pulls up the GPU metrics dashboard. Average GPU utilization: 22%. During peak hours it climbs to 31%. During off-peak it drops to 8%.

The GPU is idling. And at $3.50 per GPU-hour for an A100, the company is paying roughly$ 15,000 per month for compute that mostly does nothing.

The team traces the request flow. Requests arrive from the API gateway, hit the model server, trigger a single forward pass per request, return the result. Each forward pass takes 12ms on the GPU. Each request waits 12ms. The GPU is busy for 12ms, then idles until the next request arrives. At 200 QPS - which sounds like a lot - a new request arrives every 5ms on average, but the previous request only occupied 12ms of GPU time during that 5ms window. Worse: the GPU memory is initialized to run a batch of 64 images, but every call passes exactly 1 image. 98% of the GPU's parallel compute capacity is wasted.

The fix is batching. Instead of running one image through the model per GPU call, collect 32 requests that arrive within a 10ms window and run them together. A batch of 32 takes 18ms - only 50% longer than a single image - but processes 32× the work. GPU utilization climbs from 22% to 79%. Throughput triples. Cost per prediction drops by 67%.

This story plays out at every company that operates GPU inference at scale. The model is not the constraint - the batching strategy is.

Why This Exists - The Mismatch Between Sequential Requests and Parallel GPUs

GPUs are massively parallel processors. An A100 has 6,912 CUDA cores and can execute tens of thousands of floating-point operations simultaneously. For that parallelism to be useful, you need to give it parallel work - a batch of vectors or matrices to process at the same time.

When you run a single request through a neural network, you are running a matrix multiply between a [1, 768] input vector and a [768, 3072] weight matrix, producing a [1, 3072] output. Most CUDA cores sit idle because there is only one input vector to process. When you run 64 requests simultaneously, the input becomes [64, 768] and the output becomes [64, 3072] - the same matrix multiply processes all 64 at once, with memory bandwidth utilized nearly fully.

This is not a minor optimization. The throughput difference between batch size 1 and optimal batch size is often 20-50× for neural networks. Without batching, your GPU spend is mostly waste.

The engineering challenge is: incoming requests arrive one at a time, from many different clients, with unpredictable timing. How do you collect them into batches without making individual clients wait an unacceptably long time?

Historical Context

Early deep learning inference (2012-2016) was mostly batch processing: score all examples in the test set in one pass, get results, done. Latency was not a concern.

As ML moved into production APIs (2016-2019), the first approach was static batching: configure a fixed batch size, pad inputs to that size, always run inference on exactly N examples. Simple to implement, predictable GPU behavior. The problem: during off-peak hours, you wait until you have N requests, so p99 latency explodes.

NVIDIA introduced dynamic batching in TensorRT and Triton Inference Server around 2018-2019. The key insight was adaptive collection: gather requests for at most T milliseconds, then run whatever batch has accumulated. This bounds latency while improving GPU utilization.

The biggest recent advance came with LLMs. In 2022-2023, Orca (Yu et al., 2022) introduced continuous batching (also called in-flight batching) for autoregressive generation. The insight: in traditional batching for LLMs, the entire batch must complete generation before new requests join - if one sequence generates 1000 tokens and another generates 10, the 10-token sequence holds a slot in the batch for 990 iterations doing nothing. Continuous batching releases completed sequences immediately and fills those slots with new requests, achieving 36× higher throughput than traditional batching.

PagedAttention (Kwon et al., 2023, implemented in vLLM) further solved the memory fragmentation problem: KV cache memory is managed in fixed-size "pages" (like OS virtual memory), preventing internal fragmentation and allowing near-perfect memory utilization.

Core Concepts

Static Batching

Static batching is the baseline: always run inference on exactly $B$ examples. Inputs are padded to a fixed shape, the batch is assembled, inference runs.

$\text{Throughput} = \frac{B}{\text{latency}(B)}$

The latency of a batch grows sub-linearly with batch size (up to the point where GPU memory bandwidth saturates). This is the efficiency argument for batching: $\text{latency}(B) \ll B \cdot \text{latency}(1)$ .

The problem with pure static batching is tail latency during low-traffic periods. If $B = 32$ and only 3 requests arrive in a minute, those 3 requests wait indefinitely for 29 more to arrive.

Static batching works when:

Traffic is steady and predictable
You control both sides (offline scoring, not interactive)
You can tolerate high latency variance

Dynamic Batching

Dynamic batching adds a timeout: collect requests for up to $T_{max}$ milliseconds, then run whatever batch has accumulated. This bounds the worst-case latency at the cost of sometimes running inefficiently small batches.

The core parameters:

Max batch size $B_{max}$ : maximum requests per batch (GPU memory constraint)
Max wait time $T_{max}$ : maximum time to wait for a full batch (latency budget constraint)

# dynamic_batcher.py - minimal implementation
import asyncio
import time
from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class InferenceRequest:
    request_id: str
    input_data: bytes
    arrival_time: float = field(default_factory=time.monotonic)
    result_future: Optional[asyncio.Future] = None

class DynamicBatcher:
    def __init__(
        self,
        max_batch_size: int = 32,
        max_wait_ms: float = 10.0,
        model_fn: callable = None
    ):
        self.max_batch_size = max_batch_size
        self.max_wait_ms = max_wait_ms
        self.model_fn = model_fn
        self.pending: List[InferenceRequest] = []
        self.lock = asyncio.Lock()
        self.batch_ready = asyncio.Event()

    async def predict(self, input_data: bytes) -> dict:
        """Called by request handler - adds to queue, waits for result."""
        loop = asyncio.get_event_loop()
        future = loop.create_future()
        request = InferenceRequest(
            request_id=f"req_{id(future)}",
            input_data=input_data,
            result_future=future
        )
        async with self.lock:
            self.pending.append(request)
            if len(self.pending) >= self.max_batch_size:
                self.batch_ready.set()
        # Wait for the batch worker to process this request
        return await future

    async def batch_worker(self):
        """Background coroutine - collects requests, runs inference."""
        while True:
            # Wait for either max_batch_size requests OR timeout
            try:
                await asyncio.wait_for(
                    self.batch_ready.wait(),
                    timeout=self.max_wait_ms / 1000.0
                )
            except asyncio.TimeoutError:
                pass  # Timeout fired - process whatever we have

            async with self.lock:
                if not self.pending:
                    self.batch_ready.clear()
                    continue
                # Take up to max_batch_size requests
                batch = self.pending[:self.max_batch_size]
                self.pending = self.pending[self.max_batch_size:]
                self.batch_ready.clear()

            # Run inference on the batch (outside lock)
            inputs = [req.input_data for req in batch]
            try:
                results = self.model_fn(inputs)  # list of result dicts
                for req, result in zip(batch, results):
                    if not req.result_future.done():
                        req.result_future.set_result(result)
            except Exception as e:
                for req in batch:
                    if not req.result_future.done():
                        req.result_future.set_exception(e)

Optimal Batch Size Calculation

The right batch size balances GPU utilization against latency. The goal is to find the batch size where marginal throughput gains flatten out.

Run this profiling sweep to find the knee of the curve:

# batch_profiler.py - find optimal batch size for your model
import torch
import time
import numpy as np
from typing import Dict

def profile_batch_sizes(
    model: torch.nn.Module,
    input_shape: tuple,
    batch_sizes: list = [1, 2, 4, 8, 16, 32, 64, 128],
    warmup_iters: int = 10,
    benchmark_iters: int = 50,
    device: str = "cuda"
) -> Dict[int, dict]:
    """
    Measure latency and throughput at different batch sizes.
    Returns dict mapping batch_size -> metrics.
    """
    model.eval()
    results = {}

    for bs in batch_sizes:
        # Generate random input
        dummy_input = torch.randn(bs, *input_shape).to(device)

        # Warmup - let CUDA kernels JIT compile
        for _ in range(warmup_iters):
            with torch.no_grad():
                _ = model(dummy_input)
        torch.cuda.synchronize()

        # Benchmark
        latencies = []
        for _ in range(benchmark_iters):
            start = time.perf_counter()
            with torch.no_grad():
                _ = model(dummy_input)
            torch.cuda.synchronize()  # wait for GPU to finish
            latencies.append((time.perf_counter() - start) * 1000)

        latency_ms = np.median(latencies)
        throughput = bs / (latency_ms / 1000)
        latency_per_sample = latency_ms / bs

        results[bs] = {
            "batch_latency_ms": latency_ms,
            "throughput_samples_per_sec": throughput,
            "latency_per_sample_ms": latency_per_sample,
        }
        print(
            f"BS={bs:4d}: latency={latency_ms:.1f}ms, "
            f"throughput={throughput:.0f} samples/s, "
            f"per-sample={latency_per_sample:.2f}ms"
        )

    # Find optimal: where per-sample latency stops decreasing
    # and throughput gains diminish
    return results


# Example output for ResNet-50 on A100:
# BS=  1: latency=4.2ms,  throughput=238 samples/s,  per-sample=4.20ms
# BS=  2: latency=4.4ms,  throughput=455 samples/s,  per-sample=2.20ms
# BS=  4: latency=4.8ms,  throughput=833 samples/s,  per-sample=1.20ms
# BS=  8: latency=5.6ms,  throughput=1429 samples/s, per-sample=0.70ms
# BS= 16: latency=7.2ms,  throughput=2222 samples/s, per-sample=0.45ms
# BS= 32: latency=12.1ms, throughput=2645 samples/s, per-sample=0.38ms
# BS= 64: latency=23.0ms, throughput=2783 samples/s, per-sample=0.36ms
# BS=128: latency=45.1ms, throughput=2838 samples/s, per-sample=0.35ms
# Knee of curve: BS=32 - throughput 97% of max, latency 12ms vs 45ms for BS=128

The "optimal" batch size is at the knee: BS=32 gives 97% of maximum throughput at 12ms batch latency, while BS=128 gives only 1% more throughput at 45ms batch latency. Set $B_{max} = 32$ and $T_{max}$ to fit within your latency SLA.

Continuous Batching for LLMs

Traditional batching for LLMs fails because sequences have different lengths. If you batch 8 sequences and one generates 500 tokens while the others generate 20, the 7 short sequences finish at iteration 20 but must hold their batch slots until the long sequence completes at iteration 500.

Continuous batching (Orca, 2022) treats each iteration as an opportunity to schedule work. Slots freed by completed sequences are immediately filled with new requests.

The vLLM implementation of continuous batching with PagedAttention:

# Using vLLM for continuous batching (production LLM serving)
from vllm import LLM, SamplingParams
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
import asyncio

# Initialize vLLM engine with continuous batching
engine_args = AsyncEngineArgs(
    model="meta-llama/Llama-2-7b-chat-hf",
    tensor_parallel_size=1,          # GPUs to use
    gpu_memory_utilization=0.90,     # use 90% of GPU RAM for KV cache
    max_num_batched_tokens=8192,     # max tokens processed per iteration
    max_num_seqs=256,                # max concurrent sequences
    # PagedAttention block size (tokens per page)
    block_size=16,
)

engine = AsyncLLMEngine.from_engine_args(engine_args)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

async def generate_streaming(prompt: str, request_id: str):
    """Generate tokens with streaming - continuous batching handles concurrency."""
    results_generator = engine.generate(
        prompt,
        sampling_params,
        request_id=request_id
    )

    # Stream tokens as they are generated
    async for request_output in results_generator:
        if request_output.outputs:
            token_text = request_output.outputs[0].text
            is_finished = request_output.finished
            yield token_text, is_finished
            if is_finished:
                break


# The key insight: thousands of concurrent requests can call generate_streaming()
# simultaneously - vLLM's continuous batching scheduler interleaves them
# efficiently across GPU iterations, filling slots as sequences complete.

PagedAttention and Memory Efficiency

Standard KV cache allocation reserves the maximum sequence length upfront. A model serving up to 2048 tokens with batch size 32 reserves $32 \times 2048 \times 2 \times 768 \times 4$ bytes = 402MB just for the KV cache, regardless of actual sequence lengths.

PagedAttention allocates KV cache in fixed-size pages (e.g., 16 tokens per page), allocating new pages as the sequence grows. Memory utilization goes from ~36% (wasted on reserved-but-unused space) to ~95%.

$\text{Memory waste}_{\text{static}} = \frac{L_{max} - L_{avg}}{L_{max}} \approx 70\%$

$\text{Memory waste}_{\text{paged}} \approx \frac{\text{block\_size}}{2 \cdot L_{avg}} \approx 4\%$

Memory Implications of Batching

Batching linearly increases GPU memory consumption for activations:

$\text{Activation memory} = B \times L \times d_{model} \times \text{layers} \times \text{bytes\_per\_element}$

For a 7B parameter LLaMA model:

Sequence length $L = 512$ , $d_{model} = 4096$ , 32 layers, FP16 (2 bytes)
Per-sequence: $512 \times 4096 \times 32 \times 2 = 134$ MB (just KV cache)
Batch of 8: 1.07 GB for KV cache alone, plus 14GB for model weights
An A100 80GB can hold batch sizes up to ~40 for this model

This memory constraint sets the hard upper limit on batch size. Dynamic batching must never exceed it, or you get CUDA out-of-memory errors mid-request.

# batch_memory_estimator.py
def estimate_kv_cache_memory_gb(
    batch_size: int,
    seq_len: int,
    d_model: int,
    num_layers: int,
    num_kv_heads: int,
    d_head: int,
    bytes_per_element: int = 2  # FP16
) -> float:
    """
    Estimate KV cache memory in GB.
    Each layer stores K and V for all heads and all positions.
    """
    kv_cache_per_token = 2 * num_kv_heads * d_head * bytes_per_element
    total_bytes = batch_size * seq_len * num_layers * kv_cache_per_token
    return total_bytes / (1024 ** 3)


# LLaMA-7B: d_model=4096, 32 layers, 32 heads, d_head=128
mem = estimate_kv_cache_memory_gb(
    batch_size=16,
    seq_len=2048,
    d_model=4096,
    num_layers=32,
    num_kv_heads=32,
    d_head=128
)
print(f"KV cache: {mem:.1f} GB")  # ~16.0 GB for batch=16, seq=2048

Triton Dynamic Batching Configuration

Triton Inference Server has built-in dynamic batching that does not require custom code - just configuration:

// config.pbtxt - Triton model configuration
name: "image_classifier"
platform: "pytorch_libtorch"
max_batch_size: 64

input [
  {
    name: "input__0"
    data_type: TYPE_FP32
    dims: [ 3, 224, 224 ]
  }
]
output [
  {
    name: "output__0"
    data_type: TYPE_FP32
    dims: [ 1000 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
  max_queue_delay_microseconds: 10000  // 10ms max wait

  // Priority queue: HIGH priority requests skip ahead
  priority_queue_policy {
    priority_levels: 3
    default_priority_level: 2  // MEDIUM
  }
}

instance_group [
  {
    count: 1           // one model instance per GPU
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

With this configuration, Triton will:

Accept individual requests into a queue
Wait up to 10ms for additional requests to accumulate
Prefer batch sizes of 8, 16, or 32 (as configured)
Run inference when preferred size is reached OR timeout fires
Scatter results back to individual request handlers

Production Engineering Notes

The Latency-Throughput Tradeoff is Real

Dynamic batching always trades average latency for throughput. Even if the batch wait time $T_{max} = 10ms$ , requests arriving right after a batch is dispatched wait nearly the full 10ms before being processed. Your p99 latency will increase by approximately $T_{max}$ .

The right $T_{max}$ depends on your SLA headroom. If your SLA is 100ms and your model runs in 12ms, you have 88ms of headroom. Setting $T_{max} = 20ms$ is reasonable. If your SLA is 20ms, $T_{max} = 5ms$ or less.

Monitoring Batch Fill Rate

Track the distribution of actual batch sizes at inference time. If your mean batch size is 3 when your target is 32, the dynamic batcher is not filling batches - investigate whether traffic is too low or the timeout is too short.

# prometheus metrics for batching
from prometheus_client import Histogram, Counter

batch_size_histogram = Histogram(
    'inference_batch_size',
    'Distribution of inference batch sizes',
    buckets=[1, 2, 4, 8, 16, 32, 64, 128]
)

batch_latency_histogram = Histogram(
    'inference_batch_latency_seconds',
    'Time from first request in batch to response',
    buckets=[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]
)

queue_wait_histogram = Histogram(
    'inference_queue_wait_seconds',
    'Time request spent waiting for batch to form',
    buckets=[0.001, 0.005, 0.01, 0.025, 0.05, 0.1]
)

Common Mistakes

:::danger Running One Request Per GPU Call This is the most common mistake in naive ML serving deployments. Every request triggers a separate CUDA kernel launch, GPU memory transfer, and synchronization. At any throughput, this leaves 60-95% of GPU parallelism unused. Always implement at least static batching at the model server level, even if requests arrive one at a time. :::

:::danger Ignoring CUDA Out-of-Memory at Large Batch Sizes Setting max batch size based on "what the model supports in training" ignores serving overhead. Training uses gradient checkpointing and other memory optimizations. Inference needs space for: model weights, KV cache or activations, input/output tensors, and CUDA kernel workspace. Benchmark max batch size specifically at serving time under load. An OOM at batch size 64 mid-request drops all 64 requests simultaneously - far worse than a latency spike. :::

:::warning Padding Wastes Compute in NLP Models When batching variable-length sequences for BERT or similar models, naively padding all sequences to the longest in the batch wastes proportional compute. A batch where one sequence is 512 tokens and 31 are 32 tokens runs 16× more computation than necessary. Use sorted batching (group similar lengths together) or FlashAttention's variable-length interface to avoid padding overhead. :::

:::warning Not Accounting for Queue Wait in SLA If your SLA is "p99 latency under 100ms" and your model takes 15ms with batch size 32, you might think you have 85ms of headroom for dynamic batching. But queue wait time is additive: a request waiting 50ms for a batch + 15ms inference + 10ms network = 75ms. Track end-to-end latency from request arrival to response, not just model inference time. :::

Interview Q&A

Q1: Explain why a GPU at 20% utilization during peak load is a batching problem, not a traffic problem.

A: GPU utilization measures the fraction of time the GPU is actively computing, not idling. At 20% utilization, the GPU spends 80% of its time waiting for the next request to arrive, transfer to GPU memory, and start execution. This happens when requests are processed one at a time: each 12ms forward pass is followed by 48ms of idle time if requests arrive at 200 QPS (5ms inter-arrival). The model's arithmetic intensity - the ratio of compute to memory bandwidth - only reaches peak efficiency when the batch dimension saturates the GPU's parallel execution units. At batch size 1 on an A100, you use maybe 2-5% of available FLOPS. At batch size 32-64, you typically achieve 60-90%. The solution is dynamic batching: accumulate 10-50ms of incoming requests into a single GPU call. This is a batching problem because more traffic at the same batch size 1 would just make the utilization percentage worse (shorter idle time but same compute waste per request).

Q2: What is continuous batching and why does it matter for LLM serving?

A: Traditional LLM serving batches sequences that must all complete before the batch releases. If one sequence in a batch of 8 generates 500 tokens and the rest generate 20, the 7 short sequences hold GPU memory slots for 480 extra iterations doing nothing. Continuous batching (Orca, 2022) treats each autoregressive iteration as a scheduling opportunity. When a sequence completes generation, its slot is immediately freed and a waiting request takes its place. New requests join the running batch mid-iteration. This fundamentally changes the throughput model: instead of throughput being limited by the longest sequence in each batch, it approaches the hardware limit. vLLM's benchmark showed 36× higher throughput than naive batching using continuous batching with PagedAttention. PagedAttention solves the companion memory problem: instead of pre-allocating the maximum sequence length per slot (wasting 60-70% of KV cache memory), it allocates fixed-size "pages" on demand, reducing memory waste to under 5%.

Q3: How do you choose between max_batch_size and max_wait_time for a dynamic batcher?

A: These are complementary constraints. Start by profiling your model to find the batch size where throughput gains flatten - typically where per-sample latency stops decreasing significantly (often at BS=32-64 for image models). Set max_batch_size there. Then work backwards from your latency SLA: if SLA is 100ms and model latency at max_batch_size is 20ms, you have 80ms for queue wait plus network, so max_wait_time can be 40-60ms. If SLA is 20ms, max_wait_time must be under 5ms. The tradeoff is: higher max_wait_time improves average-case throughput (batches fill more often) but hurts tail latency (every request pays the full wait on low-traffic periods). A practical approach: set max_wait_time to 5-10% of your SLA budget, and monitor actual batch fill rate. If mean batch size is consistently below 50% of max_batch_size, your traffic may be too low for batching to help much - consider using spot instances that scale with traffic instead.

Q4: What is PagedAttention and how does it improve GPU memory utilization for LLM serving?

A: The KV cache in transformer inference stores the key and value projections for every token generated so far. Naive allocation reserves memory for the maximum possible sequence length at batch allocation time. If your model supports sequences up to 2048 tokens but the average generated sequence is 200 tokens, you waste (2048-200)/2048 ≈ 90% of your KV cache memory. This limits how many sequences you can serve concurrently. PagedAttention (from vLLM, 2023) takes inspiration from virtual memory in operating systems. The KV cache is divided into fixed-size "blocks" of, say, 16 tokens each. A sequence starts with one block allocated; as it generates more tokens, additional blocks are allocated on demand from a free-block pool. Non-contiguous blocks are fine because a block table maps logical to physical positions. Memory fragmentation drops from 60-70% (static allocation) to under 5% (paged). Freed blocks are immediately available for new sequences. This allows vLLM to serve 2-4× more concurrent sequences on the same GPU compared to non-paged implementations.

Q5: How do you handle variable-length sequences in batched NLP inference efficiently?

A: Several techniques exist, each with different tradeoffs. Sorted batching: group requests by sequence length, create batches from similar-length sequences, pad within groups only. This reduces average padding from 50% to ~10-20%. Bucket batching: define length buckets (0-64, 64-128, 128-256, 256-512) and route requests to the appropriate batcher; each bucket has its own dynamic batcher that batches only similar-length sequences. Dynamic padding: compute the max length in each batch at runtime and pad to that length rather than a fixed max - less padding than always padding to max_seq_len. FlashAttention's varlen interface: skip padding entirely by packing all sequences end-to-end in a single tensor with a cu_seqlens index that records boundaries; the attention kernel handles variable lengths natively with no wasted compute. For production transformer inference, the FlashAttention varlen approach is the most efficient - it eliminates both padding waste and the memory overhead of separate padding tokens.

The GPU Sitting Idle While You Pay for It​

Why This Exists - The Mismatch Between Sequential Requests and Parallel GPUs​

Historical Context​

Core Concepts​

Static Batching​

Dynamic Batching​

Optimal Batch Size Calculation​

Continuous Batching for LLMs​

PagedAttention and Memory Efficiency​

Memory Implications of Batching​

Triton Dynamic Batching Configuration​

Production Engineering Notes​

The Latency-Throughput Tradeoff is Real​

Monitoring Batch Fill Rate​

Common Mistakes​

Interview Q&A​