Module 02: AI Observability

AI observability is the practice of making production LLM systems transparent, measurable, and debuggable. Without observability, you cannot know if your AI is working, why it fails, or how to improve it. This module covers the full observability stack: tracing, platform-specific integrations, quality metrics, feedback loops, and automated alerting.

What You Will Learn

Lesson	Topic	Key Skill
01	Tracing LLM Applications	Capture every prompt, completion, and cost in queryable traces
02	LangSmith Deep Dive	Production observability and evals with the LangChain ecosystem
03	Langfuse	Open-source, self-hosted LLM observability
04	Phoenix by Arize	Local-first debugging with embedding drift and retrieval analysis
05	OpenTelemetry for AI	Vendor-neutral AI tracing with OTel GenAI semantic conventions
06	Quality Metrics in Production	What to measure to know if your AI is working
07	Feedback Collection	Explicit and implicit signals from users - the feedback flywheel
08	Alerting on Quality Degradation	Detect silent degradation with EWMA, z-scores, and multi-channel routing

The Observability Stack

Key Concepts

• Trace: a complete record of one LLM request - prompt, output, model, tokens, latency, cost
• Span: a single step in a multi-step pipeline (retrieval, LLM call, post-processing)
• Quality score: a numerical measure of output quality (0.0-1.0), usually from an LLM judge or user feedback
• The observability gap: traditional APM tools capture system health but not semantic quality
• Online evaluation: scoring production outputs automatically and continuously
• The feedback flywheel: user feedback → eval dataset → better prompts → better outputs → more feedback

Prerequisites

• Module 01 (LLMOps fundamentals) or equivalent experience
• Python 3.10+, anthropic SDK installed
• Basic understanding of HTTP tracing concepts (spans, traces)

Tools Covered

Platforms: LangSmith, Langfuse, Arize Phoenix, W&B Weave

Standards: OpenTelemetry GenAI semantic conventions

SDKs: langfuse, langsmith, arize-phoenix, opentelemetry-sdk

Metrics: EWMA, z-score anomaly detection, ROUGE, BERTScore