01Module 02: AI Observability - OverviewAn overview of AI observability - tracing, quality metrics, feedback collection, and alerting for production LLM applications.02Tracing LLM ApplicationsWhat tracing means for LLM apps - capturing every prompt, completion, latency, cost, and error in queryable traces. Why traditional APM fails for AI, OpenTelemetry GenAI semantic conventions, and a complete production-grade tracer implementation.03LangSmith Deep DiveMaster LangSmith for LLM observability - production tracing, dataset curation, evaluation pipelines, prompt versioning, annotation queues, and deployment gating for AI systems.04Langfuse - Open-Source LLM ObservabilityMaster Langfuse for production LLM observability - self-hosted tracing, evaluation datasets, prompt management, cost attribution by feature, and full data sovereignty for regulated industries.05Phoenix by Arize - LLM Observability with Embedding AnalysisMaster Arize Phoenix for open-source LLM observability - UMAP embedding visualization, drift detection, RAG coverage gap analysis, OpenTelemetry-native tracing, and LLM evaluation pipelines in production.06OpenTelemetry for AI SystemsApply OpenTelemetry to AI and LLM applications - GenAI semantic conventions, auto-instrumentation, OTel Collector routing, sampling strategies, context propagation through async queues, and multi-backend production setups.07Quality Metrics in Production LLM SystemsDefine, measure, and operationalize quality metrics for production LLM applications - faithfulness, answer relevance, hallucination rate, coherence, toxicity, BLEU vs LLM-as-judge, SLO definitions, and async evaluation pipelines.08Feedback Collection for LLM SystemsBuild production-grade feedback collection systems for AI products - explicit signals, implicit behavioral signals, data schemas, bias mitigation, and closed-loop improvement pipelines.09Alerting on LLM Quality DegradationBuild production alerting systems for LLM quality - threshold alerts, statistical process control, anomaly detection, deployment correlation, runbooks, and Prometheus/Grafana integration.
