Agent Observability: Full Benchmark Results

Notebook #19 — SynapseKit 1.4 vs LangChain 1.2 vs LlamaIndex Core 0.14 · local tracing only, no external service

Feature Depth Score — Local Tracing (No External Service)

SynapseKit

out of 7 features

LlamaIndex

out of 7 features

LangChain

out of 7 features

LoC to Enable Tracing & Feature Depth

Feature Matrix — What You Can See Locally

Feature	SynapseKit	LangChain	LlamaIndex
Token usage	Yes	Partial	Yes
Step latency	Yes	No	Yes
Intermediate agent steps	Yes	Yes	Yes
Tool call args + returns	Yes	Yes	Yes
Full raw LLM prompt	Yes	Yes	Yes
Retrieved documents	Yes	Yes	Yes
Zero-config enable (1–2 lines)	Yes	Yes	No
Score (out of 7)	7	5	6

Key Finding: The Latency Gap

LangChain has no local step latency

set_verbose(True) and set_debug(True) surface agent reasoning, tool calls, and raw prompts. They do not record how long any step took. Step latency — which LLM call was slow, which tool is the bottleneck, total agent run time — requires LangSmith (external service, requires API key and internet access). SynapseKit and LlamaIndex both surface timing data locally with no external dependency. This is the most significant gap for production debugging without an external platform.

Design Philosophy

SynapseKit

Explicit object

tracer = TracingMiddleware() passed to agent at construction. After run: tracer.spans returns structured list. Testable, composable, no global state.

LangChain

Global flags

set_verbose(True) is a global side effect. All subsequent LangChain objects emit to stderr. No post-run object to query. Fewest lines. Full observability requires LangSmith.

LlamaIndex

Callback manager

LlamaDebugHandler injected via Settings. Best post-run query API — typed CBEventType events, event pairs, timestamp deltas. Also global state via Settings.

Summary Table

Criterion	SynapseKit	LangChain	LlamaIndex
Fewest lines to enable	7	🏆 3	4
Local feature depth	🏆 7/7	5/7	6/7
Step latency (local)	✅ Yes	❌ No	✅ Yes
Post-run query API	Good	None (text only)	🏆 Best
Unit testable	🏆 Yes	No	Partial
Avoids global state	🏆 Yes	No	No

www.engineersofai.com · AI Letters #27 · LLM Showdown Notebook #19