Chunking Code Explorer

AI Letters #18 — LLM Showdown #9: Chunking Strategies · Select a framework to see the full pipeline
SynapseKit
LangChain
LlamaIndex
Side-by-Side
SynapseKit — 5 lines total (1 import + 4 functional)
No standalone splitter API. Chunk parameters are set directly on the Retriever. The split happens internally during r.add() — you never see or inspect the chunks. Same recursive character algorithm as LangChain under the hood.
SynapseKit — sentence-aware chunking
5 lines
from synapsekit import Retriever, InMemoryVectorStore, SynapsekitEmbeddings

# chunk params set on the Retriever — no standalone splitter API
emb = SynapsekitEmbeddings(model="all-MiniLM-L6-v2", use_gpu=False)
r   = Retriever(InMemoryVectorStore(emb),
                chunk_size=300, chunk_overlap=30)
await r.add([DOCUMENT])
What you cannot do with SynapseKit
• Inspect chunk boundaries before indexing
• Switch to a different chunking algorithm (token-based, semantic, markdown-aware)
• Log chunk size distribution for debugging retrieval quality
• Use format-aware splitting for code, Markdown, or HTML corpora
LangChain — 4 lines total (1 import + 3 functional)
The cleanest interface. Splitter is a standalone object — decoupled from the vector store, inspectable, replaceable with any of 8 built-in splitters. chunk_size = characters.
LangChain — RecursiveCharacterTextSplitter
4 lines
from langchain_text_splitters import RecursiveCharacterTextSplitter

chunks = RecursiveCharacterTextSplitter(
    chunk_size=300, chunk_overlap=30
).split_text(DOCUMENT)
Switching splitters — 1 import change
same interface
# Swap to token-based — one import change, identical method
from langchain_text_splitters import TokenTextSplitter
chunks = TokenTextSplitter(chunk_size=300, chunk_overlap=30).split_text(DOCUMENT)

# Swap to semantic — requires langchain-experimental + an embedding model
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
chunks = SemanticChunker(OpenAIEmbeddings()).split_text(DOCUMENT)
LangChain chunking strengths
• Standalone splitter — debug chunk distribution before indexing
• split_text() returns plain strings — no wrapper objects to unwrap
• 8 built-in splitters covering code (Python AST), markup (Markdown, HTML), and semantic splits
• chunk_size consistently means characters across all splitters except TokenTextSplitter
LlamaIndex — 6 lines total (2 imports + 4 functional)
Text must be wrapped in a Document object. Output is Node objects (not strings) — you extract .text. Nodes carry source metadata and relationship info. chunk_size = tokens, not characters — chunk_size=300 means ~1,200 characters.
LlamaIndex — SentenceSplitter (default)
6 lines
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import Document

nodes  = SentenceSplitter(
    chunk_size=300, chunk_overlap=30
).get_nodes_from_documents([Document(text=DOCUMENT)])
chunks = [n.text for n in nodes]
Advanced: SentenceWindowNodeParser
1 import change
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.core import Document

# Each node = 1 sentence. Surrounding sentences stored as metadata.
nodes = SentenceWindowNodeParser(
    window_size=2  # sentences on each side
).get_nodes_from_documents([Document(text=DOCUMENT)])
# At retrieval: search against precise sentence embeddings
# At generation: expand to node.metadata["window"] for full context
Advanced: HierarchicalNodeParser
production mode
from llama_index.core.node_parser import HierarchicalNodeParser
from llama_index.core import Document

# Creates 3 levels: 2048 / 512 / 128 token nodes from the same document
nodes = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
).get_nodes_from_documents([Document(text=DOCUMENT)])
# Index small (128) nodes. At query time, automerge to parent when needed.
Dimension SynapseKit LangChain LlamaIndex
Total lines 5 4 ✓ 6
chunk_size unit characters characters tokens ⚠️
Built-in splitters 2 8 9
Standalone splitter API No — opaque Yes — inspectable Yes — Node objects
Output type Internal (no access) List[str] — plain strings List[Node] — extract .text
SentenceWindow / Hierarchical No No Yes — unique
Format-aware (Markdown/Code) No Yes (Python, Markdown, HTML) Yes (Code, Markdown, JSON)
Semantic chunking No SemanticChunker (experimental) SemanticSplitterNodeParser
Cost to switch splitters Not possible 1 import line 1 import line
www.engineersofai.com