AI Letters #09 - Ship It Without Breaking Things: Evals, Observability, and Production-Safe Agents
· 3 min read
The vendor API renamed a field. The agent kept running. No error. No alert. Six days later, someone noticed the revenue reports were wrong. The agent had been silently writing null to a field that used to be called something else.
No crash. No exception. Just quiet, compounding damage.
Part 5 of 5 - the series closer: behavioral evals (not unit tests), Langfuse tracing in one callback, LangGraph interrupt/resume for HITL, input sanitization against prompt injection, cost budgets, and the 4 deployment tiers from shadow mode to full autonomy.
This letter was first published on AI Letters →
Continue reading on Substack →
Interactive Data Stories
Interactive Chart
The Road to Production-Safe AI Agents →
From Microsoft Tay (2016) to AgentBench standardization (2025) - click each event to understand the safety evolution.
Interactive Chart
The 4 Production Deployment Tiers →
Shadow mode → canary → HITL → autonomous - click each tier to see implementation, monitoring requirements, and escalation paths.
Interactive Chart
Production Agent Reliability →
Failure mode distribution, eval coverage vs incident rate, and HITL intervention rates by domain.