EngineersofAI LogoCoursesBlogResearch LabAI LettersThe LabCode BankInteractive 3DKodrEarnestJobs
Skip to main content
Interactive 3D/Embeddings in Production - Index, Latency, Cost
Production Pipeline
Documents
→
Embed (batch)
→
Index (HNSW)
→
Query (online)
→
Results
Index Memory
1.43GB
1,000,000 vectors
P99 Latency
0.1ms
HNSW, FP16
Throughput
120,000/s
batch=8
Recall@10
97%
approx search
Index Type Comparison (dim=768, FP16)
flat
Latency: 2.8msRecall: 100%
Exact search - brute force. Best recall, slowest.
HNSW
Latency: 0.1msRecall: 97%
Hierarchical NSW graph - fast ANN search, high recall.
IVF
Latency: 0.1msRecall: 93%
Inverted file index - partitions space into cells.
Cost: API vs Self-Hosted (per 1M embeddings)
Embedding API (e.g. OpenAI)
$0.13
per 1M tokens, no infra
Self-hosted (T4 GPU amortized)
$0.08
per 1M, high volume
API wins at low volume (no infra overhead). Self-hosted wins above ~5M embeddings/month. At 100M+/month, BGE/E5 on self-hosted GPU is ~5× cheaper than API.
Production Monitor - Drift Detection
Query latency P99
0.1ms ✓
Index staleness
2.3h ✓
Embedding drift
Low ✓
Cache hit rate
34% ⚠
Controls
Index Type
flat
HNSW
IVF
Embedding Dim
Quantization
2 bytes/dim · 1.8× speed
Batch Size
Batch8
Production tip: Use HNSW for online search (fast, high recall). Quantize to INT8 for 4× memory reduction with ~1% recall loss. Monitor embedding drift as your corpus evolves.

Embeddings in Production - Index, Latency, Cost - Interactive Visualization

Running embeddings in production involves three key decisions: index type (flat/HNSW/IVF), quantization (FP32/FP16/INT8), and deployment (API vs self-hosted). HNSW gives the best recall-latency tradeoff for online search. INT8 quantization reduces memory by 4x with ~1% recall loss. Self-hosted beats API cost above ~5M embeddings/month. At scale, monitoring for embedding drift (as your corpus or query distribution shifts) is critical to maintaining retrieval quality.

  • Flat index: exact search, O(n) per query, 100% recall - only for small corpora (<100k)
  • HNSW: approximate nearest neighbor, ~97% recall, sub-millisecond queries at any scale
  • IVF: inverted file partitions - fast training, tunable recall-speed tradeoff via nprobe
  • INT8 quantization: 4x memory reduction (4 bytes → 1 byte per dimension), ~1% recall drop
  • Self-hosted break-even: ~5M embeddings/month - below that, API wins on total cost

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.