:::tip 🎮 Interactive Playground Visualize this concept: Try the Sync vs Async Inference demo on the EngineersOfAI Playground - no code required. :::

Serving Architectures: REST vs gRPC vs WebSocket

The Decision That Haunts You at 3 AM

It is 11:47 PM on a Tuesday when the Slack message arrives: "image classification p99 is 340ms - SLA is 100ms - product launches in 8 hours." The ML engineer on call pulls up the metrics. The model itself runs in 12ms on average. The problem is not the model. It is everything around the model: JSON serialization taking 28ms per request, HTTP/1.1 connection overhead adding another 15ms, no connection pooling so every request pays TCP handshake cost, and the load balancer doing keep-alive poorly. The model is fast. The serving architecture is slow.

She switches the internal service-to-service calls from REST/JSON to gRPC with Protocol Buffers. The serialization cost drops from 28ms to 3ms. HTTP/2 multiplexing eliminates the per-request connection overhead. She configures connection pooling in the gRPC client. By 2 AM, p99 is at 67ms with headroom to spare. The model never changed. The protocol did.

This scenario plays out constantly at every company that scales ML. The model is the easy part - a well-tuned forward pass is deterministic and fast. The hard part is the serving layer: how requests arrive, how the model receives them, how results come back, and how thousands of concurrent requests share the same GPU backend without starving each other.

The choice of serving protocol is the most consequential architectural decision in the model serving stack. It determines the maximum throughput you can achieve, the minimum latency you can reach, whether you can support streaming inference, how much your serialization overhead will dominate at high QPS, and how painful your debugging sessions will be when something goes wrong at 3 AM.

This lesson gives you the framework to make that choice with data instead of habit. REST is not always wrong. gRPC is not always right. WebSocket is the correct answer for a specific set of problems. The goal is to understand the tradeoffs deeply enough that you never have to fix a protocol mistake at midnight.

---

Why This Exists - The Problem Before Protocols

In the early days of ML serving, models lived in monolithic applications. The Python training code was the same code that ran inference. There was no "serving" problem because the model was embedded directly in the application that used it.

As ML matured, models became separate services. The training team handed off a model artifact; the serving team deployed it. Communication between the application and the model service became a network call. And that is where the problems began.

The first-generation approach was obvious to anyone who had built web APIs: use REST with JSON. It worked. It was debuggable with curl. Existing monitoring infrastructure understood HTTP. Operations teams could reason about it. But REST/JSON has a hidden cost that only surfaces at scale: serialization. A 224×224×3 image is 150,528 float32 values - 600KB of raw data. JSON-encoding that as an array of numbers produces roughly 2.4MB of text that must be marshalled, transmitted, and unmarshalled on every request. At 10K QPS, that is 24GB/second of JSON processing - a substantial fraction of your CPU budget for work that has nothing to do with inference.

The industry needed protocols designed for high-throughput, low-latency binary data exchange between services. gRPC, built on Protocol Buffers and HTTP/2, emerged as the answer for most ML serving scenarios. But it introduced its own tradeoffs: binary encoding means you cannot debug with curl, streaming requires careful management, and browser clients cannot use gRPC directly without a proxy.

And then came LLMs. Autoregressive generation produces tokens one at a time. A user asking a question to a chatbot does not want to wait 30 seconds for the full response to buffer - they want to see words appear as they are generated. That requires a streaming protocol, and WebSocket or Server-Sent Events became the right tool for that job.

Three protocols. Three different design points. One architecture decision that you need to get right before you serve your first production request.

---

Historical Context

REST (Representational State Transfer) was defined by Roy Fielding in his 2000 PhD dissertation as an architectural style for distributed hypermedia systems. JSON became its de facto encoding during the 2000s web API boom, driven by the rise of JavaScript applications that could parse JSON natively in the browser. REST/JSON became the default for ML serving because it required zero new infrastructure - any HTTP server could expose a /predict endpoint.

gRPC was open-sourced by Google in 2015 as the evolution of their internal Stubby RPC framework. Protocol Buffers (protobuf) had been Google's internal serialization format since around 2001. The combination of protobuf encoding, HTTP/2 multiplexing, and strongly typed service definitions made gRPC the dominant choice for service-to-service communication inside large-scale systems. Triton Inference Server, TorchServe, and most production ML serving frameworks added gRPC endpoints alongside REST as their primary performance path.

WebSocket (RFC 6455, 2011) was standardized to solve the polling problem for real-time web applications. Its adoption in ML serving accelerated with the LLM era. When OpenAI released ChatGPT in 2022, streaming output over Server-Sent Events (a simpler HTTP-based alternative to WebSocket) became the expected user experience for generative models. Virtually every LLM serving framework now supports streaming as a first-class feature.

---

Core Concepts

The Three Protocols Compared

Let us start with a concrete view of what each protocol actually does at the wire level.

REST over HTTP/1.1 is stateless request-response. Each request is independent. The body is typically JSON. HTTP/1.1 allows connection reuse via keep-alive, but a single connection processes one request at a time - if you want parallelism, you need multiple connections.

gRPC over HTTP/2 uses binary Protocol Buffer encoding and HTTP/2's multiplexed streams. A single TCP connection can carry thousands of concurrent requests. Bidirectional streaming is a first-class concept. The type system enforced by .proto files catches integration errors at compile time.

WebSocket upgrades an HTTP/1.1 connection to a persistent full-duplex channel. The server can push data at any time without the client polling. It is the right tool when the server needs to stream partial results continuously, as in token-by-token LLM generation.

Serialization Cost: Where REST Loses

The critical factor for ML serving is serialization cost - the CPU time to convert your tensor data to a transmittable format and back. This is not a small cost.

For an image classification request with a 224×224×3 RGB image:

Format	Encoded size	Encode time	Decode time
JSON (base64 image)	~201 KB	~4.2 ms	~3.8 ms
JSON (float array)	~2.4 MB	~28 ms	~31 ms
Protobuf	~602 KB	~0.9 ms	~0.8 ms
Raw bytes (HTTP body)	~600 KB	~0.1 ms	~0.1 ms

At 10K QPS, JSON float array serialization costs 280 CPU-seconds per second - more than the model inference itself on many tasks. Protobuf reduces this by roughly 30x.

HTTP/2 Multiplexing: Where gRPC Wins on Concurrency

HTTP/1.1 has head-of-line blocking at the request level: a slow request on a connection blocks subsequent requests on that same connection. To achieve 100 concurrent requests, you need 100 TCP connections. Each TCP connection costs ~3ms to establish (SYN, SYN-ACK, ACK).

HTTP/2 multiplexes multiple logical streams over a single TCP connection. 1000 concurrent requests can share a single connection. This matters enormously for ML serving where a single prediction can take 50-200ms - during that time, hundreds of other requests want to start.

HTTP/1.1 (connection per request):
Conn1: [----request1----][----request5----]
Conn2: [----request2----][----request6----]
Conn3: [----request3----][----request7----]
Conn4: [----request4----][----request8----]
       (each connection = TCP handshake cost)

HTTP/2 (multiplexed, single connection):
Conn1: [r1][r2][r3][r4][r5][r6][r7][r8] (interleaved streams)
       (one TCP handshake, reused indefinitely)

Streaming Inference: Where WebSocket Wins

For autoregressive LLM generation, the model produces output token by token. A 200-token response at 30 tokens/second takes 6.7 seconds to generate completely. With request-response protocols (REST or gRPC unary), the user waits 6.7 seconds to see anything. With streaming, the first token arrives in ~100ms - the user experience transforms from waiting to watching.

WebSocket enables the server to push tokens as they are generated without the client polling. Server-Sent Events (SSE) is a simpler alternative that works over HTTP/1.1 - it is unidirectional (server to client only) but sufficient for most LLM streaming use cases.

---

Production Server Architectures

TorchServe

TorchServe (from Meta/AWS) is the reference serving framework for PyTorch models. It exposes both a REST Management API (port 8081) and a REST Inference API (port 8080), with a gRPC endpoint added in v0.5.

TorchServe's core abstraction is the handler - a Python class that implements preprocessing, inference, and postprocessing. The framework manages worker processes, batching, and request routing.

# custom_handler.py - TorchServe handler
import torch
import torchvision.transforms as transforms
from ts.torch_handler.base_handler import BaseHandler
from PIL import Image
import io
import base64

class ImageClassificationHandler(BaseHandler):
    def initialize(self, context):
        """Load model once at startup - called per worker process."""
        self.manifest = context.manifest
        properties = context.system_properties
        model_dir = properties.get("model_dir")

        # Load model to GPU if available
        self.device = torch.device(
            "cuda:" + str(properties.get("gpu_id"))
            if torch.cuda.is_available() else "cpu"
        )
        self.model = self._load_model(model_dir)
        self.model.to(self.device)
        self.model.eval()

        # Define preprocessing pipeline
        self.transform = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]
            )
        ])
        self.initialized = True

    def preprocess(self, data):
        """Called once per batch - data is a list of request bodies."""
        images = []
        for row in data:
            # Accept base64-encoded image or raw bytes
            image_data = row.get("data") or row.get("body")
            if isinstance(image_data, str):
                image_data = base64.b64decode(image_data)
            image = Image.open(io.BytesIO(image_data)).convert("RGB")
            images.append(self.transform(image))
        # Stack into a single batch tensor
        return torch.stack(images).to(self.device)

    def inference(self, data):
        """Run forward pass - data is preprocessed batch tensor."""
        with torch.no_grad():
            return self.model(data)

    def postprocess(self, inference_output):
        """Convert logits to class predictions for each request in batch."""
        probabilities = torch.softmax(inference_output, dim=1)
        top5_probs, top5_indices = probabilities.topk(5, dim=1)
        results = []
        for probs, indices in zip(top5_probs, top5_indices):
            results.append({
                "predictions": [
                    {"class": int(idx), "probability": float(prob)}
                    for idx, prob in zip(indices, probs)
                ]
            })
        return results

Triton Inference Server

NVIDIA Triton is the production choice when you need maximum performance, multi-model serving, or non-Python runtimes. Triton supports TensorRT, ONNX, PyTorch (TorchScript), TensorFlow, and Python backends on a single server instance.

Triton natively supports both gRPC (port 8001) and HTTP REST (port 8000) with the same backend. Use gRPC for internal microservice calls; REST for external-facing APIs or debugging.

# triton_client.py - calling Triton via gRPC
import tritonclient.grpc as grpcclient
import numpy as np

def run_inference(server_url: str, model_name: str, image_np: np.ndarray):
    """
    image_np: numpy array of shape (H, W, 3) uint8
    Returns: class probabilities
    """
    client = grpcclient.InferenceServerClient(url=server_url)

    # Preprocess: resize, normalize, add batch dimension
    # (assumed preprocessed to [1, 3, 224, 224] float32)
    input_data = preprocess(image_np)  # returns shape (1, 3, 224, 224)

    # Define input tensor
    inputs = [
        grpcclient.InferInput("input__0", input_data.shape, "FP32")
    ]
    inputs[0].set_data_from_numpy(input_data)

    # Define expected output
    outputs = [
        grpcclient.InferRequestedOutput("output__0")
    ]

    # Run inference - gRPC call
    response = client.infer(
        model_name=model_name,
        inputs=inputs,
        outputs=outputs,
    )

    probabilities = response.as_numpy("output__0")
    return probabilities  # shape (1, num_classes)


def preprocess(image_np: np.ndarray) -> np.ndarray:
    """Resize to 224x224, normalize with ImageNet stats."""
    from PIL import Image
    import numpy as np

    image = Image.fromarray(image_np).resize((224, 224))
    arr = np.array(image, dtype=np.float32) / 255.0
    mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
    std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
    arr = (arr - mean) / std
    # HWC to CHW, add batch dimension
    arr = arr.transpose(2, 0, 1)[np.newaxis, ...]
    return arr

REST Serving with FastAPI (for Simplicity)

For lower-traffic scenarios or when you need maximum flexibility in request handling, a FastAPI server with explicit batching logic is a reasonable choice:

# fastapi_server.py - custom REST inference server
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import asyncio
import torch
import numpy as np
from typing import List
import time

app = FastAPI()

# Global model (loaded once at startup)
model = None
model_lock = asyncio.Lock()

class PredictionRequest(BaseModel):
    image_b64: str  # base64-encoded image bytes
    request_id: str

class PredictionResponse(BaseModel):
    request_id: str
    class_id: int
    confidence: float
    latency_ms: float

@app.on_event("startup")
async def load_model():
    global model
    model = torch.jit.load("model.pt")
    model.eval()
    if torch.cuda.is_available():
        model = model.cuda()

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    start = time.perf_counter()

    try:
        # Decode and preprocess
        import base64
        from PIL import Image
        import io

        image_bytes = base64.b64decode(request.image_b64)
        image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
        tensor = preprocess_image(image)

        # Inference - use lock to avoid GPU memory corruption
        # in concurrent requests on single GPU
        async with model_lock:
            with torch.no_grad():
                output = model(tensor)

        probs = torch.softmax(output[0], dim=0)
        class_id = int(probs.argmax())
        confidence = float(probs[class_id])

        latency_ms = (time.perf_counter() - start) * 1000
        return PredictionResponse(
            request_id=request.request_id,
            class_id=class_id,
            confidence=confidence,
            latency_ms=latency_ms
        )

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

:::warning Avoid the Global Lock Pattern The model_lock pattern above prevents concurrent requests from corrupting GPU state, but it serializes all inference - killing throughput. The correct solution is dynamic batching (Lesson 02). Use this pattern only for development or very-low-traffic deployments. :::

The Sidecar Pattern for Zero-Downtime Model Updates

Updating a model while serving live traffic is one of the trickiest operational problems. The sidecar pattern solves it by running the new model version as a sidecar container in the same Kubernetes pod, using a lightweight proxy to manage traffic routing during updates.

The update sequence:

1. New model version downloads and loads in the sidecar process (no traffic yet)
2. Sidecar warms up: a small batch of shadow requests runs through the new model
3. Proxy shifts traffic gradually: 5% → 25% → 50% → 100%
4. If error rate or latency spikes, proxy rolls back instantly
5. Old model process gracefully shuts down after all in-flight requests complete

---

Protocol Selection Decision Framework

Benchmark: REST vs gRPC at Scale

Here is what you can expect in real systems (these are representative numbers, not guarantees - your mileage will vary by hardware, payload size, and model):

Metric	REST/JSON	gRPC/Protobuf	Improvement
Serialization (224×224 image)	28 ms	0.9 ms	~31×
p50 latency (10K QPS)	45 ms	18 ms	~2.5×
p99 latency (10K QPS)	340 ms	87 ms	~3.9×
Throughput (single connection)	120 QPS	1,800 QPS	~15×
CPU for serialization	35%	4%	~8.8×
Payload size (batch of 32)	76 MB	19 MB	4×

The gRPC advantage grows with payload size and request volume. For text-only requests (e.g., NLP classification with short strings), the difference is smaller - perhaps 2× rather than 15×.

---

Concrete Implementation: gRPC Service Definition

// inference.proto - service definition
syntax = "proto3";

package inference.v1;

// Service definition - compiled to generate client/server stubs
service InferenceService {
    // Unary: single request, single response
    rpc Predict (PredictRequest) returns (PredictResponse);

    // Server streaming: single request, stream of response chunks
    rpc PredictStream (PredictRequest) returns (stream PredictChunk);

    // Bidirectional streaming: continuous request/response pairs
    rpc PredictBatch (stream PredictRequest) returns (stream PredictResponse);
}

message PredictRequest {
    string request_id = 1;
    string model_name = 2;
    bytes image_data = 3;       // raw image bytes - no base64 overhead
    int32 top_k = 4;            // return top-k predictions
    map<string, string> metadata = 5;
}

message PredictResponse {
    string request_id = 1;
    repeated Prediction predictions = 2;
    float inference_latency_ms = 3;
}

message Prediction {
    int32 class_id = 1;
    string class_name = 2;
    float confidence = 3;
}

message PredictChunk {
    string request_id = 1;
    string token = 2;           // for LLM streaming
    bool is_final = 3;
}

# grpc_server.py - Python gRPC server implementation
import grpc
from concurrent import futures
import inference_pb2
import inference_pb2_grpc
import torch
import time

class InferenceServicer(inference_pb2_grpc.InferenceServiceServicer):
    def __init__(self, model_path: str):
        self.model = torch.jit.load(model_path)
        self.model.eval()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model.to(self.device)

    def Predict(self, request, context):
        """Unary RPC - one request, one response."""
        start = time.perf_counter()

        try:
            # Decode raw bytes directly - no base64 overhead
            tensor = decode_image_bytes(request.image_data)
            tensor = tensor.to(self.device)

            with torch.no_grad():
                logits = self.model(tensor)

            probs = torch.softmax(logits[0], dim=0)
            top_probs, top_indices = probs.topk(request.top_k or 5)

            latency_ms = (time.perf_counter() - start) * 1000

            return inference_pb2.PredictResponse(
                request_id=request.request_id,
                predictions=[
                    inference_pb2.Prediction(
                        class_id=int(idx),
                        confidence=float(prob)
                    )
                    for idx, prob in zip(top_indices, top_probs)
                ],
                inference_latency_ms=latency_ms
            )
        except Exception as e:
            context.set_code(grpc.StatusCode.INTERNAL)
            context.set_details(str(e))
            return inference_pb2.PredictResponse()

    def PredictStream(self, request, context):
        """Server-streaming RPC - for LLM token generation."""
        # Simulate token-by-token generation
        for token in generate_tokens(request):
            if context.is_active():
                yield inference_pb2.PredictChunk(
                    request_id=request.request_id,
                    token=token,
                    is_final=False
                )
            else:
                break  # Client disconnected - stop generating
        yield inference_pb2.PredictChunk(
            request_id=request.request_id,
            token="",
            is_final=True
        )


def serve(model_path: str, port: int = 50051):
    # Thread pool for handling concurrent requests
    server = grpc.server(
        futures.ThreadPoolExecutor(max_workers=10),
        options=[
            ('grpc.max_send_message_length', 50 * 1024 * 1024),   # 50MB
            ('grpc.max_receive_message_length', 50 * 1024 * 1024),
            ('grpc.keepalive_time_ms', 30000),    # ping every 30s
            ('grpc.keepalive_timeout_ms', 5000),  # wait 5s for pong
        ]
    )
    inference_pb2_grpc.add_InferenceServiceServicer_to_server(
        InferenceServicer(model_path), server
    )
    server.add_insecure_port(f'[::]:{port}')
    server.start()
    server.wait_for_termination()

---

Production Engineering Notes

Connection Pool Sizing

For gRPC clients, connection pool size is critical. Too few connections: requests queue up. Too many: you waste resources and risk overwhelming the server.

A reasonable starting formula: pool_size = (max_concurrent_requests * avg_latency_s) / target_utilization

For 500 concurrent requests at 50ms average latency targeting 70% connection utilization: pool_size = (500 * 0.05) / 0.7 ≈ 36 connections.

Load Balancing for gRPC

Standard L4 load balancers (AWS NLB, GCP TCP LB) work poorly with gRPC because they balance at the connection level, not the request level. Since HTTP/2 multiplexes many requests on one connection, connection-level balancing can send all traffic to a single backend.

Solutions:

• L7 load balancers (Envoy, NGINX, Istio): understand HTTP/2 stream semantics
• Client-side load balancing: each client maintains connections to all backends and routes requests round-robin
• gRPC-lb: gRPC's built-in name resolution can return multiple addresses; combined with round-robin policy, achieves request-level balancing

WebSocket for LLM Streaming

For production LLM streaming, prefer Server-Sent Events (SSE) over WebSocket for most use cases. SSE is simpler (HTTP-based, no upgrade handshake), automatically reconnects on disconnect, works through HTTP proxies and load balancers without special configuration, and is sufficient because LLM streaming is unidirectional (server to client).

Use WebSocket when you need full-duplex communication: streaming audio/video input to the model while receiving streaming output simultaneously.

---

Common Mistakes

:::danger Using REST for High-QPS Internal Services If your model service is called by another microservice - not a browser - and you are seeing more than 200-300 QPS, REST/JSON serialization overhead will dominate your latency budget. Every engineer who has profiled a slow ML serving stack has found JSON marshalling as a top-5 CPU consumer. Switch to gRPC early; retrofitting it after launch is painful. :::

:::danger Forgetting gRPC Load Balancing Deploying gRPC behind a standard TCP load balancer and wondering why one pod gets 90% of traffic while others are idle is one of the most common operational mistakes with gRPC. HTTP/2 long-lived connections defeat round-robin at the connection level. Use Envoy, Istio, or client-side load balancing. :::

:::warning Not Handling Client Disconnection in Streaming For LLM streaming endpoints, the client frequently disconnects before the generation completes (user navigates away, request times out). If your server does not detect disconnection and stop generating, you waste GPU compute generating tokens nobody will read. Always check context.is_active() in gRPC or handle ConnectionClosedError in WebSocket handlers. :::

:::warning Ignoring TLS for Internal Services It is tempting to skip TLS for service-to-service gRPC calls inside a private VPC. Do not. Modern security requirements (SOC2, zero-trust networking) require encryption in transit everywhere. Configure mutual TLS (mTLS) from day one - retrofitting it means rotating credentials across all clients and servers simultaneously. :::

---

Interview Q&A

Q1: Why would you choose gRPC over REST for a model serving endpoint? What are the concrete benefits?

A: The primary benefits are serialization efficiency and connection multiplexing. Protocol Buffers encode the same data in roughly 30% of the space compared to JSON, and the encode/decode CPU cost is about 30× lower. At 10K QPS with large payloads (images, audio), this difference can be the majority of your request budget. HTTP/2 multiplexing means one TCP connection handles thousands of concurrent requests without head-of-line blocking, which matters when model inference takes 20-100ms per request. Additionally, .proto schema definitions catch API contract violations at compile time rather than at runtime. The main downside is operational complexity: gRPC is harder to debug without specialized tooling (grpcurl, grpc-gateway), and browsers cannot call gRPC directly without a proxy like Envoy or grpc-gateway.

Q2: How does WebSocket differ from SSE for LLM streaming, and when would you pick each?

A: Both are streaming protocols, but SSE is unidirectional (server to client) while WebSocket is bidirectional. For LLM token streaming, SSE is usually the better choice: it is simpler (HTTP-based, no protocol upgrade), automatically reconnects on failure, passes through HTTP proxies and load balancers without configuration, and most LLM use cases only need server-to-client streaming. Use WebSocket when you need simultaneous input and output streaming - for example, voice-to-voice AI where you are streaming audio to the server while receiving generated audio back. WebSocket also has lower per-message overhead for high-frequency bidirectional use cases.

Q3: Describe the sidecar pattern for zero-downtime model updates.

A: The sidecar pattern runs the new model version as a second process (or container) in the same pod as the current version. A lightweight proxy sits in front of both and controls traffic routing. The update sequence is: (1) the new model downloads and loads - no traffic yet; (2) a warm-up phase runs synthetic or shadow requests through the new model to initialize GPU memory and JIT compilation caches; (3) the proxy gradually shifts traffic - 1% → 5% → 25% → 50% → 100% - monitoring error rate and latency at each step; (4) the old model stays running until all in-flight requests complete, then gracefully shuts down. If any step shows degradation, the proxy instantly routes 100% back to the old model. This avoids the pod restart cycle that would drop connections and requires re-establishing GPU state.

Q4: How do you handle load balancing for gRPC services correctly?

A: Standard L4 load balancers route at the TCP connection level, but HTTP/2 multiplexes many requests per connection - so a long-lived connection from one client ends up pinned to one backend forever. The solution depends on your infrastructure: in Kubernetes with Istio or Linkerd, the service mesh handles HTTP/2-aware L7 load balancing automatically. With Envoy, you configure the cluster as http2_protocol_options and it routes at the stream level. For non-service-mesh environments, client-side load balancing works: each client resolves all backend addresses and round-robins requests across them. The gRPC libraries (Go, Java, Python) have built-in support for this via the grpc.experimental.gevent channel or custom name resolvers.

Q5: A REST-based image classification service handles 500 QPS fine, but at 2,000 QPS the CPU is 100% and latency spikes. What is likely the bottleneck and how do you fix it?

A: The most likely bottleneck is JSON serialization. At 500 QPS the serialization overhead is manageable; at 2,000 QPS it saturates CPUs. Profile with py-spy or cProfile - you will likely see the JSON encoder/decoder consuming 40-60% of CPU. The fix has two parts: (1) switch the internal data encoding from JSON to Protocol Buffers or raw bytes, which cuts serialization cost by 20-30×; (2) if the endpoint must remain REST-compatible (for external clients), switch the internal communication (from API gateway to model server) to gRPC and keep REST only at the edge. You can also preprocess the serialization by using orjson instead of the standard json module - it is 3-10× faster for float-heavy payloads - but this is a stopgap, not a solution at higher scale.

Q6: What is head-of-line blocking in HTTP/1.1, and why does HTTP/2 solve it?

A: In HTTP/1.1, a TCP connection can only process one request at a time. If a slow request occupies the connection, subsequent requests must wait, even if they are much faster. This is head-of-line blocking. In practice, HTTP/1.1 clients open multiple parallel connections (typically 6 per domain) to work around this, but this multiplies TCP handshake and congestion control overhead. HTTP/2 introduces the concept of streams - independent logical channels multiplexed over a single TCP connection. Request 100 can start while requests 1-99 are still in flight. A slow request on stream 5 does not block stream 42. For ML serving where requests vary from 5ms (cached, simple input) to 500ms (long generation, complex input), this is critical: you do not want fast requests blocked behind slow ones on the same connection.