9 docs tagged with "ai-observability"

Build production alerting systems for LLM quality - threshold alerts, statistical process control, anomaly detection, deployment correlation, runbooks, and Prometheus/Grafana integration.

Build production-grade feedback collection systems for AI products - explicit signals, implicit behavioral signals, data schemas, bias mitigation, and closed-loop improvement pipelines.

Master Langfuse for production LLM observability - self-hosted tracing, evaluation datasets, prompt management, cost attribution by feature, and full data sovereignty for regulated industries.

Master LangSmith for LLM observability - production tracing, dataset curation, evaluation pipelines, prompt versioning, annotation queues, and deployment gating for AI systems.

An overview of AI observability - tracing, quality metrics, feedback collection, and alerting for production LLM applications.

Apply OpenTelemetry to AI and LLM applications - GenAI semantic conventions, auto-instrumentation, OTel Collector routing, sampling strategies, context propagation through async queues, and multi-backend production setups.

Master Arize Phoenix for open-source LLM observability - UMAP embedding visualization, drift detection, RAG coverage gap analysis, OpenTelemetry-native tracing, and LLM evaluation pipelines in production.

Define, measure, and operationalize quality metrics for production LLM applications - faithfulness, answer relevance, hallucination rate, coherence, toxicity, BLEU vs LLM-as-judge, SLO definitions, and async evaluation pipelines.

What tracing means for LLM apps - capturing every prompt, completion, latency, cost, and error in queryable traces. Why traditional APM fails for AI, OpenTelemetry GenAI semantic conventions, and a complete production-grade tracer implementation.