AI Letters #19 - The BM25 Test: One Framework Silently Fails

BM25 ships in one framework, requires a hidden install in another, and silently fails at runtime in the third. Same class name, three very different experiences.

Pure vector search has a blind spot. Exact-match queries - model names, function names, version numbers, proper nouns - embed poorly. The query "GPT-4o" and the document "GPT-4o" don't always produce similar vectors. BM25 does not have this problem. It matches terms, weighs them by rarity, and returns the right document.

Production RAG systems almost always use hybrid search: BM25 for precision on exact matches, vector search for semantic recall, reciprocal rank fusion to merge them. The question of whether BM25 ships out of the box is not academic. It determines whether your pipeline works on day one or fails at 2am in a customer demo.

We tested all three frameworks on an identical task: index five documents, run a BM25 query, get top-3 results. One framework's BM25Retriever class is in its package but silently throws a ModuleNotFoundError at runtime unless you've separately installed a library it doesn't list as a dependency.

Interactive Chart

Install Path by Framework →

What you have to install before BM25 works - extra packages, silent dependencies, and integration packages across all three frameworks.

Code Explorer

BM25 Pipeline Code by Framework →

Full BM25 index + query code side by side - imports, setup, and retrieval call annotated for each framework.

Data

LoC, Extra Installs, and Result Overlap →

Lines of code breakdown, extra packages required, and ranked result comparison for identical query across all three frameworks.

What We Measured

Task: Index 5 documents, run a BM25 query, return top-3 results. Identical corpus and query across all three frameworks.

Metric	What it captures
Extra packages needed	Pip installs beyond the base framework install
Lines of code	Import + functional lines to build and query a BM25 index
Result quality	Top-3 docs returned for an identical keyword query

Frameworks: SynapseKit 1.4, LangChain 1.2, LlamaIndex Core 0.14. Kaggle CPU.

The Install Story

Before a single line of BM25 code runs, you need the right packages.

Framework    Base install              Extra needed              Behavior
──────────────────────────────────────────────────────────────────────────
SynapseKit   pip install synapsekit    none                      Works immediately
LangChain    pip install langchain     pip install rank-bm25     Silent runtime fail if missing
             langchain-community
LlamaIndex   pip install llama-index-core  pip install           ImportError at import time
                                      llama-index-retrievers-bm25

LangChain's behavior is the most dangerous. BM25Retriever lives in langchain-community. The import succeeds. The class is there. But when you call BM25Retriever.from_texts(), it raises ModuleNotFoundError: No module named 'rank_bm25' - a runtime error, not an import error. Your code passes linting, passes static analysis, and fails in production.

LlamaIndex fails at import time - from llama_index.retrievers.bm25 import BM25Retriever - which is the honest failure mode. You find out immediately.

SynapseKit declares rank-bm25 as a core dependency in its pip metadata. It installs with the base package. Nothing extra to do.

The Code

LangChain - 3 lines (1 import + 2 functional):

from langchain_community.retrievers import BM25Retriever

r       = BM25Retriever.from_texts(DOCS, k=3)
results = [doc.page_content for doc in r.invoke(QUERY)]

The cleanest BM25 API across all three. from_texts() takes a list of strings, invoke() returns Document objects. Three lines total.

SynapseKit - 8 lines (2 imports + 6 functional):

from synapsekit.retrieval import HybridSearchRetriever, Retriever, InMemoryVectorStore
from synapsekit.embeddings import SynapsekitEmbeddings

emb    = SynapsekitEmbeddings(model="all-MiniLM-L6-v2", use_gpu=False)
r      = Retriever(InMemoryVectorStore(emb))
hybrid = HybridSearchRetriever(r, bm25_weight=1.0, vector_weight=0.0)
hybrid.add_documents(DOCS)
await r.add(DOCS)
results = await hybrid.retrieve(QUERY, top_k=3)

SynapseKit's BM25 is hybrid-first. There is no standalone keyword retriever - BM25 lives inside HybridSearchRetriever with bm25_weight=1.0. This means you initialise an embedding model and a vector store even when you only want keyword search. The embedding model never runs (weight is 0), but the object must exist. Eight lines for something that should be three.

LlamaIndex - 9 lines (3 imports + 6 functional):

from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core import Document, Settings
from llama_index.core.node_parser import SentenceSplitter

Settings.llm = None
Settings.embed_model = None
nodes   = SentenceSplitter(chunk_size=512).get_nodes_from_documents(
              [Document(text=d) for d in DOCS])
r       = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=3)
results = [n.text for n in r.retrieve(QUERY)]

Three imports, two explicit None assignments to suppress LLM/embedding warnings, and a node parsing step before the retriever can be initialised. Nine lines, most of it overhead suppression.

The Results

Query: "How does BM25 compare to TF-IDF?"

Rank   SynapseKit                    LangChain                     LlamaIndex
───────────────────────────────────────────────────────────────────────────────
#1     TF-IDF weights terms...       RAG feeds retrieved passages   TF-IDF weights terms...
#2     RAG feeds retrieved passages  BM25 is a probabilistic...     BM25 is a probabilistic...
#3     Hybrid search combines...     Hybrid search combines...      Hybrid search combines...

Result overlap (Jaccard): 0.50 across all pairs (2/3 shared each)

All three retrieve the same 3 documents from a 5-document corpus - they differ only on ranking order. That is expected: all three use BM25Okapi from the rank_bm25 library under the hood. Different tokenisation details shift the ranking slightly, but the relevant documents are the same.

The result quality question is a non-issue for BM25. What matters is whether it runs at all.

What This Means for Engineers

1. LangChain's silent runtime failure is a production hazard. A ModuleNotFoundError inside from_texts() - not at import time - means it passes every pre-deploy check that doesn't exercise the retrieval path. Add rank-bm25 to your requirements file explicitly, always.

2. SynapseKit's hybrid-first design costs you 5 extra lines for pure keyword search. If you only want BM25, you're initialising an embedding model that never runs. The zero-install story is real; the ergonomics for standalone BM25 are not great.

3. LlamaIndex's explicit install is the honest design. A separate package for BM25 means the base install stays small. The tradeoff is one more pip install you have to know about - but at least it fails at import time, not at 2am in production.

4. In practice, you want hybrid search, not pure BM25. Pure BM25 as a benchmark is useful; as a production retriever it leaves semantic recall on the table. The real question is which framework makes hybrid search (BM25 + vector + RRF) easiest to configure - that's the next benchmark.

5. All three use the same BM25 algorithm. BM25Okapi from rank_bm25 is the de facto standard implementation in Python. The retrieval quality differences you see in production are almost never about the BM25 implementation - they're about tokenisation, stemming, and stopword handling that sits on top of it.

The Corollary Most People Miss

The install story matters more than the LoC story for BM25.

LangChain wins on lines of code (3 vs 8 vs 9). But a 3-line retriever that silently fails in production is worth less than an 8-line retriever that works. The ergonomics cost of SynapseKit's hybrid-first design is real - you shouldn't have to initialise embeddings to do keyword search - but at least it doesn't fail on you.

LlamaIndex's approach is the cleanest philosophically: BM25 is a separate concern, it lives in a separate package, the failure mode is immediate and visible. The ergonomics in code are the worst, but the operational behaviour is the most honest.

Design philosophy comparison:
────────────────────────────────────────────────────────────
SynapseKit    BM25 bundled, hybrid-first API
              ✓ zero extra installs
              ✗ cannot do standalone BM25 without embedding overhead

LangChain     BM25 class included, dependency external
              ✓ cleanest API (3 lines)
              ✗ silent runtime failure if rank-bm25 not installed

LlamaIndex    BM25 in separate package, explicit install
              ✓ honest failure mode (import error, not runtime error)
              ✗ most verbose (9 lines + Settings suppression)

Three Things Worth Doing This Week

1. Audit your requirements file. If you use LangChain's BM25Retriever, confirm rank-bm25 is in your requirements.txt or pyproject.toml. The import succeeds without it; the runtime doesn't.

2. Run a hybrid retrieval experiment on your existing RAG pipeline. Add BM25 alongside your vector search, fuse with reciprocal rank fusion, measure precision@3 on 20 representative queries. Most teams see 10–25% improvement on exact-match queries with no change to the embedding model.

3. Read the Kaggle notebook. Full reproducible code, the live ranked results, and the result overlap analysis: LLM Showdown #10 - Built-in BM25

---

BM25 is 35 years old and still in production at Google, Elasticsearch, and every search system that handles exact-match queries. The question was never whether to use it. The question was whether your framework ships it without surprises. One does. One requires a hidden install. One fails silently at runtime. Now you know which is which.

Engineers of AI

Read more: www.engineersofai.com

If this was useful, forward it to one engineer who should be reading it.