Courses Blog Research Lab AI Letters The Lab Code Bank Interactive 3DKodr Earnest Jobs

Interactive 3D/REST vs gRPC for ML Serving

REST vs gRPC for ML Serving

Medium (32 samples) - Unary RPC - Internal Service

Payload Size Comparison

REST (JSON)

47.0 KB

HTTP/1.1 · text

gRPC (Protobuf)

13.0 KB

HTTP/2 · binary · 72% smaller

gRPC is 72% smaller - saves 34.0 KB per request

Latency Breakdown (ms)

Serialization

REST

0.5ms

gRPC

0.1ms

Deserialization

REST

0.5ms

gRPC

0.1ms

Network

REST

0.4ms

gRPC

0.1ms

REST total: 1.3ms

gRPC total: 0.3ms

Speedup: 4.4×

REST - JSON format

{
  "model": "fraud-v2",
  "instances": [{
    "user_id": "u_8473",
    "amount": 249.99,
    "features": [0.12, -0.5,
      1.34, 0.78, ...]
  }]
}

gRPC - Protobuf schema

// fraud.proto
message InferRequest {
  string model = 1;
  repeated Instance instances = 2;
}
message Instance {
  string user_id = 1;
  float amount = 2;
  repeated float features = 3;
}
// Binary wire: ~60% smaller than JSON

Feature Matrix

Feature	REST	gRPC
Protocol	HTTP/1.1	HTTP/2
Serialization	JSON (text)	Protobuf (binary)
Browser support	✓	⚠ grpc-web only
Streaming	⚠ SSE/WebSocket	✓
Schema enforcement	⚠ OpenAPI opt-in	✓
Code generation	⚠ manual/OpenAPI	✓
Human readable	✓	✗
Binary protocol	✗	✓
Multiplexing	✗	✓
Header compression	✗	✓

Recommendation for Internal Service

REST

Acceptable but verbose. Extra JSON serialization overhead adds up at scale (millions of requests/hour).

gRPC

Strongly preferred for service-to-service. Protobuf reduces payload 40-70%. HTTP/2 multiplexing enables connection reuse.

Payload Size

RPC Mode

Use Case

Display

Serialization breakdown

Rule of Thumb

Internal ML services: gRPC. External/browser-facing APIs: REST. LLM token streaming: gRPC server-streaming or SSE.

REST vs gRPC for ML Serving - Interactive Visualization

REST and gRPC are the two dominant protocols for ML model serving APIs. REST over HTTP/1.1 with JSON is human-readable, universally supported, and easy to debug. gRPC over HTTP/2 with Protobuf is 40–70% smaller in payload, supports connection multiplexing, and enables server-side streaming for LLM token output. For internal ML services handling high QPS, gRPC's lower serialization overhead and binary encoding yield measurable latency improvements. For external or browser-facing APIs, REST remains the practical default due to tooling support and debuggability.

Protobuf binary encoding is 40–70% smaller than JSON for the same float array data - directly reduces network and serialization cost
HTTP/2 multiplexing: gRPC can serve multiple concurrent requests over a single connection - fewer TCP handshakes at scale
LLM streaming: gRPC server-side streaming sends tokens incrementally; REST requires SSE or WebSocket for the same pattern
Rule of thumb: internal services use gRPC, public/browser APIs use REST - start REST for debugging, migrate to gRPC at high QPS

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.