AI Letters #17 - The PDF Test: Which Framework Makes You Learn a New API?

The question isn't how many lines it takes to load a PDF. It's how many new concepts you need to know.

Three frameworks. Same task: take a PDF off disk, build a queryable index, answer a question. We measured lines of code and wall-clock indexing time.

The line counts are almost beside the point. What matters is this: two of the three frameworks require you to learn a loader API that didn't exist in the string version. One doesn't.

Interactive

Code Walk-Through: String vs PDF →

See exactly what changed (and what didn't) when we switched from string input to a PDF file - line by line across all three frameworks.

Framework Explorer

Full Pipeline Code by Framework →

Select a framework to see its complete PDF RAG pipeline, annotated with which lines are imports vs functional code.

Data

Lines of Code + Indexing Time →

Stacked LoC breakdown (imports vs functional) and wall-clock indexing time across all three frameworks on Kaggle CPU.

What We Measured

Same pipeline, new input format:

Step           Description
──────────────────────────────────────────────────
1. Load        Read a PDF file from disk
2. Chunk       Split into overlapping passages
3. Embed       Encode with all-MiniLM-L6-v2
4. Index       Build an in-memory vector store
5. Query       Retrieve top-k for a test question

Document: A 3-page technical PDF about RAG systems (generated with fpdf2 - reproducible). Embeddings: all-MiniLM-L6-v2 via sentence-transformers, same for all three (fair comparison). Timing: Median of 3 runs. Covers PDF load → chunk → embed → index build. LLM query excluded (network latency, not framework overhead).

The Code, Side by Side

SynapseKit - 4 lines (1 import, 3 functional):

from synapsekit import RAG

rag = RAG(model="gpt-4o-mini", api_key=API_KEY)
rag.add("rag_guide.pdf")
answer = rag.ask_sync(QUERY)

LlamaIndex - 8 lines (3 imports, 5 functional):

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm         = OpenAI(model="gpt-4o-mini", api_key=API_KEY)
Settings.embed_model = OpenAIEmbedding(api_key=API_KEY)
documents = SimpleDirectoryReader(input_files=["rag_guide.pdf"]).load_data()
index     = VectorStoreIndex.from_documents(documents)
answer    = index.as_query_engine(similarity_top_k=5).query(QUERY)

LangChain - 14 lines (5 imports, 9 functional):

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

pages  = PyPDFLoader("rag_guide.pdf").load()
chunks = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50).split_documents(pages)
vs     = FAISS.from_documents(chunks, OpenAIEmbeddings(api_key=API_KEY))
chain  = RetrievalQA.from_chain_type(
             llm=ChatOpenAI(model="gpt-4o-mini", api_key=API_KEY),
             retriever=vs.as_retriever(search_kwargs={"k": 5})
         )
answer = chain.invoke({"query": QUERY})["result"]

What Changed vs Hello RAG (#3)

In benchmark #3, the input was a plain string. Here it's a PDF. Exact delta:

Framework    #3 LoC   #8 LoC   Delta   What changed
──────────────────────────────────────────────────────────────
SynapseKit      4        4       0     Nothing. rag.add() is format-agnostic.
LangChain      13       14      +1     Added PyPDFLoader import. Removed
                                       Document wrapping (loader handles it).
LlamaIndex      9        8      -1     Swapped Document(text=...) for
                                       SimpleDirectoryReader(...).load_data()

SynapseKit's delta is zero. The rag.add() method accepts strings, file paths, and URLs - PDF detection is automatic. No new import, no new API surface, no new concept to learn.

The Indexing Time Numbers

Framework    Indexing time (median 3 runs)
──────────────────────────────────────────
LangChain    2.8 s
SynapseKit   3.1 s
LlamaIndex   3.4 s

The spread is 21% across all three. The embedding model is the dominant cost. Framework overhead - PDF parsing, object construction, internal bookkeeping - is real but secondary.

What This Actually Measures

The conceptual surface area grows with LangChain and LlamaIndex when you switch from strings to files. You have to know that document loaders are a separate abstraction layer.

LangChain ships ~50 document loaders (Confluence, Notion, GitHub, SharePoint, S3, and more). The consistent .load() → List[Document] API across all of them is real value for heterogeneous data pipelines. LlamaIndex's SimpleDirectoryReader auto-detects file types - pass it a directory with PDFs, Word docs, and plain text and it routes each correctly without you specifying the format. That's genuine value for mixed corpora.

SynapseKit              LangChain / LlamaIndex
────────────────────    ────────────────────────────────
rag.add(anything)       One loader class per file type
No new concepts         New abstraction: document loaders
PDF is transparent      PyPDFLoader, SimpleDirectoryReader,
                        S3FileLoader, ConfluenceLoader...

What This Means for Engineers

1. API surface area compounds. Every new input format in LangChain/LlamaIndex means a new loader to find and remember. SynapseKit's single rag.add() handles everything - until you need control over something it hides.

2. Indexing time is dominated by embeddings. The framework overhead is noise on a 3-page PDF. The number you care about is your embedding model's throughput on your hardware and document volume.

3. Explicitness has maintenance value. LangChain's verbose pipeline makes every transformation visible. When retrieval degrades in production, you know exactly where to add logging.

4. LlamaIndex's auto-detection is underrated. One loader call for a mixed-format corpus is genuinely useful in production. The LoC savings vs LangChain are modest; the operational convenience is not.

5. The next benchmark is what matters. Chunking strategy (#9) drives retrieval quality more than any of these metrics.

Three Things Worth Doing This Week

1. Test your framework's error on a missing file. rag.add("nonexistent.pdf") - does it fail fast with a clear message or deep in the embedding stack?

2. Check whether your loader preserves page metadata. LangChain's PyPDFLoader attaches page numbers. LlamaIndex preserves source metadata. SynapseKit abstracts it. Whether you need page-level attribution determines which matters.

3. Read the Kaggle notebook. Full code, reproducible timing methodology, and the live retrieval demo (no API key needed for retrieval): LLM Showdown #8 - RAG from a PDF

---

The PDF test didn't change the ranking. It confirmed the abstraction philosophy: SynapseKit hides the format layer, LangChain exposes it, LlamaIndex exposes it with auto-detection. Week 2 continues with chunking strategies - where the frameworks diverge in ways that actually affect retrieval quality.

Engineers of AI

Read more: www.engineersofai.com

If this was useful, forward it to one engineer who should be reading it.