BM25 Evidence Dashboard — AI Letters #19

Top-3 Results — Query: "How does BM25 compare to TF-IDF?"

Rank	SynapseKit	LangChain	LlamaIndex
#1	TF-IDF weights terms by frequency…	RAG feeds retrieved passages…	TF-IDF weights terms by frequency…
#2	RAG feeds retrieved passages…	BM25 is a probabilistic…	BM25 is a probabilistic…
#3	Hybrid search combines…	Hybrid search combines…	Hybrid search combines…

All three return the same 3 documents from a 5-document corpus — only ranking order differs. All use BM25Okapi from rank_bm25 under the hood. Tokenisation details shift order, not content.

Result Overlap (Jaccard Similarity)

0.50

SynapseKit
vs LangChain
2/3 shared docs

0.50

SynapseKit
vs LlamaIndex
2/3 shared docs

0.50

LangChain
vs LlamaIndex
2/3 shared docs

Jaccard = |A ∩ B| / |A ∪ B| — The same 3 docs appear across all pairs (Doc2 BM25, Doc3 hybrid, Doc5 TF-IDF) with rank ordering shifted. Result quality is a non-issue for BM25. What matters is whether it runs at all.