AI Letters #16 - Week 1 Scorecard: Which LLM Framework Wins on Developer Experience?

The simplest framework won on developer experience. That's the expected result. The interesting part is which benchmark it lost - and why.

Six weeks of benchmarks. Six dimensions of the developer experience you have on day one of a new project. One aggregate score.

SynapseKit wins 5 of 6. The one it loses - raw import speed - is a deliberate architectural choice by LangChain, not a gap in capability. Understanding why reveals more about framework design than any of the wins.

Interactive Chart

Benchmark Series Walk-Through →

Click through all six notebooks - raw numbers and key finding per benchmark.

Framework Explorer

Design Philosophy by Framework →

Select a framework to see its full benchmark profile and the design decisions behind each number.

Scorecard

Week 1 Final Standings →

Full points table, per-benchmark bar charts, and normalized performance comparison across all six dimensions.

What We Measured

Six dimensions of the out-of-the-box developer experience:

Benchmark         Metric                   Direction
──────────────────────────────────────────────────────
#1 Cold Start     Import time (s)          Lower is better
#2 Dep Hell       Transitive packages      Lower is better
#3 Hello RAG      Lines of code            Lower is better
#4 Memory         Idle RAM (MB)            Lower is better
#5 Provider Swap  Lines changed per swap   Lower is better
#6 Error Quality  Clarity score (0–5)      Higher is better

Scoring: 3 pts for 1st, 2 pts for 2nd, 1 pt for 3rd. Ties split the point pool.

The Six Numbers

Cold Start - Import time:

LangChain:  0.16 s  ← 1st
SynapseKit: 0.27 s
LlamaIndex: 1.40 s

Dependency Hell - Transitive packages:

SynapseKit:  2  (numpy, rank-bm25)  ← 1st
LangChain:  32
LlamaIndex: 60

Hello RAG - Lines to build a full RAG pipeline:

SynapseKit:  4 lines  ← 1st
LlamaIndex:  9 lines
LangChain:  13 lines

Memory Footprint - Idle RSS at import:

SynapseKit:  38 MB  ← 1st
LangChain:   72 MB
LlamaIndex:  98 MB

Provider Switching - Changes per swap:

SynapseKit: 1 line changed  ← 1st
LangChain:  3 changes
LlamaIndex: 3 changes

Error Messages - Clarity/actionability score:

SynapseKit:  3.47  ← 1st
LangChain:   3.20
LlamaIndex:  2.73

The Scorecard

Benchmark      SynapseKit  LangChain  LlamaIndex  Winner
─────────────────────────────────────────────────────────
#1 Cold Start       2          3           1       LangChain
#2 Dep Hell         3          2           1       SynapseKit
#3 Hello RAG        3          1           2       SynapseKit
#4 Memory           3          2           1       SynapseKit
#5 Provider         3         1.5         1.5      SynapseKit
#6 Errors           3          2           1       SynapseKit
─────────────────────────────────────────────────────────
TOTAL              17.0       11.5         7.5

What the Loss Reveals

LangChain's lazy loading is a real architectural decision. Most of its weight defers until first use - that's why import is 0.16 s despite shipping hundreds of integrations. The 32-package install is still there, tree-shaking doesn't apply, but the critical path stays fast.

SynapseKit is eager. What you see is what you get. The 0.11 s gap is real but not what matters - your first LLM call costs 500 ms.

LangChain / LlamaIndex         SynapseKit
────────────────────────────   ─────────────────────────────
Large surface area             Small surface area
Lazy loading, fast init        Eager loading, transparent
Per-provider packages          provider= string, no installs
Flexible escape hatches        Opinionated defaults

What This Means for Engineers

1. DX is not capability. These benchmarks measure day one. LlamaIndex's built-in chunking strategies, evaluation pipelines, and observability don't appear anywhere here. Week 2 capability benchmarks may tell a different story.

2. The dependency count compounds. 60 packages means 60 points of version conflict, 60 packages to audit for security, 60 entries in your lock file. For teams with strict dependency governance, this is a constraint, not a preference.

3. Lines of code is a proxy for abstraction level. 4 vs 13 lines for the same pipeline means the framework made 9 decisions for you. Whether those are the right decisions depends on your requirements.

4. Silent failures are never acceptable. LlamaIndex's no-error-at-init pattern - build a full index, get an auth error on first query - has no defense. One isinstance check costs 2 lines. There's no tradeoff here.

5. Lazy loading is smart in specific contexts. Monorepos, Lambda functions, multi-module applications - anywhere cold start on the critical path matters. For standalone LLM scripts, the benefit disappears.

The Thing Most People Miss

Week 1 metrics measure the selection phase. The maintenance phase is where most cost lives.

The dependency count compounds monthly. The error message quality determines debugging time at 2am. The lines of code metric matters once per feature; the mental model it encodes matters every day.

These six numbers capture one week. Week 2 starts the capability benchmarks: PDF ingestion, chunking strategies, hybrid search, streaming, conversation memory. The developer experience leader and the capability leader may not be the same framework.

Three Things Worth Doing This Week

1. Run pip show <your-framework> | grep Requires. Count the transitive dependencies. Ask whether you need all of them.

2. Test your framework's init failure modes. Wrong API key. Wrong model name. No credentials. Time how long the error takes to understand.

3. Read the Kaggle notebook. All six benchmarks, raw output cells, methodology, and scoring code: LLM Showdown #7 - Week 1 Scorecard

---

The framework with the best developer experience scores won because it's smaller and more opinionated. Real advantage on day one. The question for week four is whether the opinions are still the right ones - or whether you've outgrown them.

Engineers of AI

Read more: www.engineersofai.com

If this was useful, forward it to one engineer who should be reading it.