Retrieval Strategy Comparison: Answer Correctness on Multi-Hop QA
Multi-hop QA requires combining information from 2+ documents. Answer correctness measured by human evaluation + exact match. Higher retrieval quality compounds across steps in agent pipelines.
Key Finding
HyDE (Hypothetical Document Embeddings) achieves a 42% relative improvement over no retrieval and a 27% improvement over naive top-k. The mechanism: generating a hypothetical answer before embedding aligns the query vector with the document embedding space, reducing the style mismatch between short queries and long technical documents. Reranking on top of HyDE pushes correctness to 81%, at the cost of one additional LLM inference call per retrieval.
31%
LLM only, no retrieval baseline
+161%
Relative gain: reranking + HyDE vs no retrieval
27%
HyDE improvement over naive top-k
Source: Gao et al. (2023), "Precise Zero-Shot Dense Retrieval without Relevance Labels" (HyDE paper, arXiv:2212.10496); Trivedi et al. (2022) "MuSiQue: Multihop Questions via Single-hop Question Composition"; additional multi-hop QA evaluations from BEIR benchmark. Numbers normalized for illustration.
Memory Latency vs Capacity
Each memory option sits at a different point in the latency/capacity tradeoff space. Bubble size indicates relative operational cost. In-context has zero latency but hard token limits; vector stores trade latency for unlimited capacity.
Key Finding
For most agent workloads, ChromaDB local (15ms) and pgvector (30ms) represent the sweet spot — fast enough for real-time agent loops (which are bottlenecked by LLM latency at 500-2000ms anyway), unlimited capacity, and zero managed infrastructure cost. Pinecone is justified at billion-vector scale. Full re-embedding (400ms+) should only occur on corpus updates, never per-query.
0ms
In-context access latency
Note on Cost Bubbles
Bubble size represents relative operational cost: in-context is free at retrieval time but pays per-token at inference; Redis requires an always-on server; Pinecone charges per read/write unit at scale. ChromaDB and pgvector are effectively free at moderate scale (<10M vectors).
Source: ChromaDB benchmark reports (2024); pgvector HNSW benchmarks (Supabase Engineering, 2024); Pinecone latency SLAs; internal measurements. Latency values are p50 on commodity hardware, 768-dimensional vectors, n_results=5.
Context Window Utilization in Production Agents
What actually fills context windows in real agentic workloads. Measured across multi-step research and coding agents running 15-25 steps. Numbers represent average % of total context window budget consumed at step 15.
Tool results (accumulated)
41%
Available for new input
8%
Key Finding
The biggest context budget consumer is accumulated tool results (41%), not conversation history. Every tool call returns a result that gets appended to the context. After 15 steps, a search agent may have accumulated 50,000+ tokens of tool output. The naive solution is to increase context window size. The correct solution is to summarize and evict tool results into semantic memory as soon as they've been processed, keeping only the extracted facts in context.
41%
Tool results: largest context consumer
8%
Available for new input at step 15
20%
Avg headroom saved by result summarization
Practical Implication
If you are running agents that take more than 10-12 steps, build a result-summarization step into your loop. After each tool call: extract key facts, store them in semantic memory, replace the raw tool output in context with a 2-3 sentence summary. This can recover 15-25% of context budget and meaningfully extend how long agents can operate before hitting the window limit.
Source: Analysis of agent traces from LangSmith (LangChain), internal agent monitoring, and published case studies from Cognition AI and Devin (2024). Percentages are averages at step 15 across research and coding agent workloads. System prompt assumes ~2K tokens; tool schemas assume 4 tools with full JSON definitions.