AI Letters #18 - The Chunking Test: Two Frameworks Are Identical, One Is Not

How you split documents determines what your retriever finds. Most tutorials spend two lines on this. They shouldn't.

Every RAG tutorial reaches the chunking step and sprints past it. "Split into chunks of 500 characters with 50 overlap - done." The code runs. The demo works. The demo is not production.

The split you choose affects embedding quality, retrieval precision, and whether your LLM gets enough context to say something useful. Chunking is not configuration. It's architecture.

We ran all three frameworks against the same document with identical parameters. The line counts came out nearly equal. The chunk outputs did not. One framework's default splitter interprets chunk_size=300 as tokens, not characters - producing 2 chunks averaging 986 characters each instead of 12 chunks averaging 163 characters. Same parameter name, different semantics.

Interactive Chart

Splitter Inventory by Framework →

All built-in splitters across SynapseKit, LangChain, and LlamaIndex - what ships out of the box and what each one is for.

Code Explorer

Full Chunking Code by Framework →

Select a framework to see the complete chunking pipeline with line-by-line annotation: imports, splitter config, and output.

Data

Chunk Count, Size Distribution, and LoC →

Live chunk output from all three frameworks on identical input - count, average size, max size, and size histogram side by side.

What We Measured

Task: Split a 1,972-character document about RAG systems into chunks. Parameters: chunk_size=300, chunk_overlap=30, sentence-aware splitter for each framework. Metrics:

Metric	What it captures
Built-in splitter count	How many strategies ship out of the box
Lines of code	How much code to configure sentence-aware chunking
Chunk output	Count, avg size, size distribution from identical input

Frameworks: SynapseKit 1.4, LangChain 1.2, LlamaIndex Core 0.14. Kaggle CPU environment.

The Splitter Inventory

Before measuring LoC, count what each framework ships.

SynapseKit - 2 splitters:

• RecursiveTextChunker - recursive character splitting (default)
• TokenChunker - token-count-based splitting

LangChain - 8 splitters:

• RecursiveCharacterTextSplitter - recursive character splitting (recommended default)
• CharacterTextSplitter - single-separator character splitting
• TokenTextSplitter - token-count splitting
• SentenceTransformersTokenTextSplitter - sentence-transformer token splitting
• MarkdownTextSplitter - markdown-header-aware splitting
• PythonCodeTextSplitter - Python AST-aware splitting
• HTMLSectionSplitter - HTML section-aware splitting
• SemanticChunker - embedding-based semantic splitting (langchain-experimental)

LlamaIndex - 9 splitters:

• SentenceSplitter - sentence-aware splitting (default)
• TokenTextSplitter - token-count splitting
• CodeSplitter - language-aware code splitting
• MarkdownNodeParser - markdown-header-aware splitting
• JSONNodeParser - JSON-structure-aware splitting
• SentenceWindowNodeParser - sentence with surrounding context window
• HierarchicalNodeParser - multi-level hierarchical chunks
• SemanticSplitterNodeParser - embedding-based semantic splitting
• TopicNodeParser - topic-model-based splitting

Two vs eight vs nine. The headline number is misleading though - what matters is whether the advanced splitters solve problems you'll actually encounter. We'll come back to this.

The Code, Side by Side

SynapseKit - 5 lines (1 import, 4 functional):

from synapsekit import Retriever, InMemoryVectorStore, SynapsekitEmbeddings

emb = SynapsekitEmbeddings(model="all-MiniLM-L6-v2", use_gpu=False)
r   = Retriever(InMemoryVectorStore(emb),
                chunk_size=300, chunk_overlap=30)
await r.add([DOCUMENT])

No standalone splitter API. Chunk parameters live on the Retriever. If you want to inspect chunks before indexing, you can't - the split is opaque.

LangChain - 4 lines (1 import, 3 functional):

from langchain_text_splitters import RecursiveCharacterTextSplitter

chunks = RecursiveCharacterTextSplitter(
    chunk_size=300, chunk_overlap=30
).split_text(DOCUMENT)

The cleanest interface of the three. Splitter is a standalone object, inspectable, composable. You can run it before indexing, log the output, swap it for any other splitter without touching the rest of your pipeline.

LlamaIndex - 6 lines (2 imports, 4 functional):

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import Document

nodes  = SentenceSplitter(
    chunk_size=300, chunk_overlap=30
).get_nodes_from_documents([Document(text=DOCUMENT)])
chunks = [n.text for n in nodes]

Two imports instead of one. Text must be wrapped in a Document object. Output is Node objects, not strings - you extract .text yourself. More verbose, but the Node carries metadata (source, position, relationships) that becomes useful downstream.

Line count totals: SynapseKit 5 / LangChain 4 / LlamaIndex 6. Effectively tied.

The Chunk Output Surprise

Same document. Same chunk_size=300. Same chunk_overlap=30. Here is what each framework produced:

Framework    Chunks   Avg size (chars)   Max size (chars)
──────────────────────────────────────────────────────────
SynapseKit      12         163               254
LangChain       12         163               254
LlamaIndex       2         986              1481

SynapseKit and LangChain are identical - SynapseKit uses the same recursive character algorithm under the hood. LlamaIndex produced 2 chunks averaging 986 characters each.

The reason: LlamaIndex's SentenceSplitter interprets chunk_size as tokens, not characters. chunk_size=300 means 300 tokens, which is roughly 1,200 characters. On a 1,972-character document, that yields 2 chunks - not the 12 you'd expect if you assumed character-based sizing.

This is not a bug. It is the documented behavior. But it is also the most common source of confusion when engineers switch frameworks mid-project. You copy the parameters from a LangChain tutorial, paste them into LlamaIndex, and your chunk distribution changes by an order of magnitude without a single error message.

                    chunk_size=300 means...
                    ┌──────────────────────────────────┐
SynapseKit          │ 300 characters                   │
LangChain           │ 300 characters                   │
LlamaIndex          │ 300 tokens (~1,200 characters)   │
                    └──────────────────────────────────┘

Same document, chunk_size=300, overlap=30:
SynapseKit/LangChain:  [163][163][163][163][163][163][163][163][163][163][163][163]
LlamaIndex:            [──────────────────1481──────────────────][───490───]

What LlamaIndex's Advanced Splitters Actually Do

The splitter count difference is real, but two of LlamaIndex's nine entries represent strategies with no equivalent in the other two frameworks.

SentenceWindowNodeParser: Stores each sentence as an individual node. Attaches the surrounding sentences as a metadata window (configurable, default: 1 sentence each side). At retrieval time, you search against precise single-sentence embeddings - high precision. At generation time, you expand the retrieved node to its window - adequate context. The result is retrieval that finds the exact sentence you need without diluting the embedding with surrounding text. Neither LangChain nor SynapseKit has a built-in equivalent.

HierarchicalNodeParser: Creates three levels of nodes from the same document: large (2048 tokens), medium (512), small (128). Small nodes are indexed for retrieval. When retrieval returns too many small nodes from the same parent (configurable threshold), they are "automerged" into the parent chunk before being sent to the LLM. You get the precision of small chunks with the coherence of large ones. This is a production technique; the LlamaIndex documentation attributes meaningful accuracy gains to it on multi-hop questions.

Switching between LlamaIndex's splitters costs one line - the import:

# One import change. Everything else stays identical.
from llama_index.core.node_parser import SentenceSplitter           # default
from llama_index.core.node_parser import SentenceWindowNodeParser   # precision mode
from llama_index.core.node_parser import HierarchicalNodeParser     # production mode

What This Means for Engineers

1. Never copy chunk parameters across frameworks without checking the unit. chunk_size=500 in LangChain is characters. In LlamaIndex it is tokens. Verify once, avoid a silent quality regression.

2. SentenceWindowNodeParser is worth understanding even if you don't use LlamaIndex. The pattern - retrieve at sentence granularity, generate with window context - is implementable in any framework manually. LlamaIndex just makes it one import.

3. HierarchicalNodeParser solves a real production problem. When retrieval returns five fragments of the same paragraph as separate nodes, your LLM is reading five partial views of the same text. Automerging collapses them into the parent. This is not theoretical - it matters on documents with repeated cross-references.

4. SynapseKit's 2 splitters are a constraint when you need format-aware splitting. If your corpus includes Markdown docs, Python files, and HTML pages, you need a splitter that understands structure. LangChain and LlamaIndex have these. SynapseKit does not.

5. LangChain's standalone splitter API is the most flexible for debugging. Because chunking is decoupled from the vector store, you can log chunk distributions before committing to an indexing run. In production, that observability pays back quickly.

The Corollary Most People Miss

The line counts say LangChain is cheapest (4 lines), LlamaIndex most expensive (6), SynapseKit in the middle (5). That is the wrong frame.

The actual cost comparison is: how many lines does it take to switch splitters when your default stops working?

For LlamaIndex: one line (the import). All nine splitters share the same get_nodes_from_documents() interface.

For LangChain: also roughly one line - all splitters expose .split_text() or .split_documents().

For SynapseKit: you cannot switch splitters. The chunking algorithm is not exposed. You take what the Retriever does internally, or you switch frameworks.

Initial LoC favors LangChain. Iteration cost favors LlamaIndex. Lock-in risk penalizes SynapseKit.

Three Things Worth Doing This Week

1. Print your chunk distribution before indexing. [len(c) for c in chunks] - histogram it. If 20% of your chunks are under 50 characters, your splitter is cutting at punctuation. If 20% exceed your embedding model's token limit, they're being silently truncated.

2. Test LlamaIndex's SentenceWindowNodeParser on one of your existing retrievers. The interface is one import and one additional retrieval step. If your current precision is poor, sentence-level retrieval with window expansion frequently outperforms standard chunking without any change to the embedding model.

3. Read the Kaggle notebook. Full reproducible code for all three frameworks, live chunk outputs, and the size distribution charts: LLM Showdown #9 - Chunking Strategies

---

Chunking determines what your retriever can find. The tutorials that sprint past it in two lines are the same tutorials whose RAG demos fall apart on real documents. The split is not configuration. It is the first decision that determines whether your retrieval is precise or lucky.

Engineers of AI

Read more: www.engineersofai.com

If this was useful, forward it to one engineer who should be reading it.