AI Letters #23 - The RAG Scorecard: Six Benchmarks, Three Frameworks, One Clear Pattern

"Batteries-included beats fully-composable on conciseness every time. Fully-composable beats batteries-included on control every time. You just have to know which problem you're solving."

Six notebooks. Six benchmarks. Three frameworks measured on the same RAG workloads, back to back, reproducible on Kaggle.

Week 1 of the LLM Showdown covered setup overhead: environment spin-up, indexing speed, basic retrieval, reranking, evaluation harnesses, and the Week 1 scorecard. SynapseKit won that one 15–7–8 (SK–LC–LI).

Week 2 went deeper into the RAG stack: PDF ingestion, chunking strategies, BM25 availability, hybrid search RRF, streaming time-to-first-token, and conversation memory. Same methodology. 3-2-1 points for rank 1-2-3 across each benchmark, ties split.

The results are not a surprise if you've been paying attention. But the magnitude of the gap on some dimensions is.

Interactive Chart

Two-Week Journey →

All 14 notebooks, cumulative standings after each week, with notebook-by-notebook breakdown of winners and key findings.

Interactive Explorer

Benchmark Explorer →

Click each of the 6 benchmarks to explore raw values, methodology, and what the result actually means in production.

Evidence Dashboard

Full Scorecard Dashboard →

Complete points heatmap, stacked benchmark breakdown, raw values, and two-week cumulative standings in one view.

Here is what the data says.

What the Scorecard Shows

Week 2 final points:

Framework     #8   #9   #10  #11  #12  #13   Total
──────────────────────────────────────────────────
SynapseKit   3.0  1.0  3.0  2.0  3.0  3.0   15.0
LlamaIndex   2.0  3.0  1.5  1.0  2.0  2.0   11.5
LangChain    1.0  2.0  1.5  3.0  1.0  1.0    9.5

SynapseKit wins 4 of 6 benchmarks. LangChain wins 1. LlamaIndex wins 1. Same pattern as Week 1, except LlamaIndex and LangChain swap second and third depending on the dimension.

Two-week cumulative:

SynapseKit:  15 (W1) + 15 (W2) = 30
LlamaIndex:   8 (W1) + 11.5 (W2) = 19.5
LangChain:    7 (W1) + 9.5 (W2)  = 16.5

The Evidence - Benchmark by Benchmark

#8 - RAG from PDF (lines of code)

SynapseKit loads a PDF into a retrieval pipeline in 7 lines. LangChain needs 13. LlamaIndex needs 11. The LangChain number is not lazy code - it requires a PyPDFLoader, a RecursiveCharacterTextSplitter, a vector store, and a retriever. Each is a separate abstraction. SynapseKit wraps all of that into one RAGPipeline(pdf="...") call.

Winner: SynapseKit. Margin: nearly 2x.

#9 - Chunking Strategies (built-in splitter count)

LlamaIndex wins this cleanly: 9 built-in splitters vs LangChain's 7 vs SynapseKit's 4. The two that matter are SentenceWindowNodeParser (retrieves surrounding sentences, not just the matched chunk) and HierarchicalNodeParser (builds a tree of chunks at different granularities). Neither exists in SynapseKit or LangChain. If your retrieval quality depends on chunk context, LlamaIndex is the right tool.

Winner: LlamaIndex. Not close.

#10 - Built-in BM25 (extra packages required)

SynapseKit bundles rank_bm25 as a core dependency. LangChain and LlamaIndex both require you to install an extra package (rank-bm25 and llama-index-retrievers-bm25 respectively) before BM25 is available. Zero vs one extra pip install. It sounds trivial. At deployment time in a locked environment, it is not.

Winner: SynapseKit.

#11 - Hybrid Search RRF (configurability score)

LangChain wins this one, and it deserves to. EnsembleRetriever accepts an arbitrary list of retrievers and per-retriever weights. You can combine three different retrievers with custom weighting in a single constructor call. LlamaIndex's hybrid search has no weight control - it applies RRF with fixed parameters. SynapseKit sits in the middle: two-retriever support, fixed alpha weighting.

Winner: LangChain. Score: 5/5 vs SK's 4/5 vs LI's 3/5.

#12 - Streaming TTFT (median framework overhead, ms)

SynapseKit:   0.001 ms
LlamaIndex:   0.184 ms
LangChain:    0.236 ms

All three are sub-millisecond. SynapseKit's async generator adds the least overhead. But read the caveat in the takeaway section - this benchmark's winner does not matter in production.

Winner: SynapseKit. Winner that matters: nobody.

#13 - Conversation Memory (lines of code to add memory)

SynapseKit: 4 lines. LlamaIndex: 6 lines. LangChain: 12 lines.

LangChain's RunnableWithMessageHistory requires a store object, a getter function, a session ID, and LCEL wiring before the history is injected. SynapseKit exposes it as one constructor parameter: memory=True. The gap is 3x.

Winner: SynapseKit.

What This Means for Engineers

1. SynapseKit wins on conciseness in every dimension where conciseness is the metric. PDF loading, BM25, memory wiring - all 3-4x fewer lines. If you are prototyping or building internal tooling where developer velocity matters more than edge-case flexibility, this is the path.

2. LangChain wins when you need fine-grained control over retrieval composition. Hybrid search with custom weights across three retrievers is a real use case - recommendation engines, multi-index RAG, domain-specific blending. EnsembleRetriever handles this; SynapseKit's fixed alpha does not.

3. LlamaIndex wins when chunking quality is the bottleneck. If you're working with long technical documents, legal text, or anything where retrieved chunk context matters, SentenceWindowNodeParser and HierarchicalNodeParser are not features - they are the reason to use LlamaIndex.

4. The TTFT result is noise. Sub-millisecond framework overhead against a real LLM API that adds 300–2000ms of network latency. Do not let this benchmark influence your framework choice.

5. Week 3 is where it gets interesting. Agents, tool calling, multi-agent orchestration - this is where the architectures diverge most sharply. SynapseKit's agent layer is newer. LangChain's is battle-tested. LlamaIndex's is designed for data-heavy agentic workflows. The conciseness advantage SynapseKit holds in RAG may not hold in agents.

The Corollary Most People Miss

The benchmarks SynapseKit loses are the ones that reveal its design tradeoff. Fewer splitters means less chunking flexibility. Fixed hybrid search alpha means less retrieval control. No persistent memory backends (yet) means you own the storage problem.

SynapseKit is fast to write. It is not yet flexible to extend.

LangChain is slow to write. It is extremely flexible to extend - the entire LCEL composability model exists precisely to let you plug in arbitrary steps without rewriting the framework.

Neither is wrong. They are optimised for different constraints. The mistake is reaching for LangChain's full composability when you are building a standard RAG pipeline that SynapseKit already handles in 7 lines. The inverse mistake is reaching for SynapseKit when you need custom retrieval logic that requires LangChain's EnsembleRetriever.

Know which problem you have before you pick the tool.

Three Things Worth Doing This Week

1. Run the notebooks yourself - all 6 are reproducible on Kaggle CPU. Fork LLM Showdown #8 through #13. Swap in your own documents and LLM endpoint. The numbers in your environment may differ from ours.

2. Audit your chunking strategy. Most RAG implementations use RecursiveCharacterTextSplitter with default chunk size because it is the default. Check if SentenceWindowNodeParser or a sliding window approach would improve your retrieval precision. Run a quick eval on 20 representative queries before assuming it does not matter.

3. Profile your own framework overhead end-to-end. Not the TTFT micro-benchmark we ran - the full round trip: query → retrieve → generate → first token to your user. That number is what your users experience. Framework choice is usually not in the top three factors.

Week 3 covers agents. ReAct loops, function calling, tool libraries, multi-agent coordination, tracing, and error handling. The scorecard will look different. Check back.

Engineers of AI

Read more: www.engineersofai.com

If this was useful, forward it to one engineer who should be reading it.