:::tip 🎮 Interactive Playground Visualize this concept: Try the LLM Observability demo on the EngineersOfAI Playground - no code required. :::

Phoenix by Arize - LLM Observability with Embedding Analysis

The Problem That Traces Could Not Show

Three weeks after launching a RAG-powered documentation assistant, user satisfaction drops from 4.2 to 2.8 out of 5. The engineering team has LangSmith set up and running. Individual traces are visible - every request, every retrieved chunk, every model response. The team spends forty minutes inspecting traces, sampling failing conversations at random. Five traces. Ten traces. Twenty. Nothing is obviously wrong. The model is answering questions correctly about core product features. It is failing on questions about recent updates. But none of the individual failing traces reveals why - each one looks like a plausible retrieval miss.

The problem is not in any single trace. It is a population-level problem - something systematic across thousands of interactions that only becomes visible when you look at the entire distribution at once. No individual trace can surface a structural gap. You need to see all of them simultaneously.

The team spins up Phoenix. They load a week of query embeddings - every user question vectorized and rendered in 2D using UMAP projection. In two minutes, the root cause is unmistakable: there are two dense clusters of user queries sitting in a region of embedding space where there are zero retrieved document embeddings nearby. Questions about the new API endpoints - released exactly three weeks ago - are clustering together in a void. The retrieval index was never updated with the new documentation. The model is not hallucinating, it is not reasoning poorly, it simply has nothing relevant to retrieve for an entire category of user questions. No individual trace showed this. The UMAP shows the structural gap in the knowledge base instantly.

The team reindexes the new documentation, redeployes, and monitors Phoenix over the next 24 hours. The dark cluster of low-quality interactions fills in with green. Satisfaction recovers within a day. Phoenix found in two minutes what 40 minutes of trace inspection had not surfaced - not because traces are bad, but because this class of problem is fundamentally invisible at the individual-trace level.

This is the core insight behind Phoenix: traces answer "what happened in this interaction," but UMAP embedding visualization answers "what is happening across all interactions." Both questions matter. Only Phoenix answers the second one.

What Is Phoenix - Open-Source, Local-First, OTEL-Native

Phoenix is an open-source LLM and ML observability platform built by Arize AI. It is designed specifically for the class of problems that trace-level tools cannot surface: population-level failure patterns, embedding drift, retrieval coverage gaps, and distribution shift. It runs locally with a single pip install, requires no external service to get started, and uses OpenTelemetry as its native instrumentation protocol - meaning it is composable with every other OTel-compatible observability backend.

Arize AI built Phoenix drawing directly on enterprise ML monitoring infrastructure developed before the LLM era. Arize was already doing embedding drift detection and slice-level performance analysis for recommendation systems and NLP classifiers when the LLM wave arrived. The team recognized that the failure modes of production RAG and agent systems - coverage gaps, query drift, cluster-level quality degradation - were structurally identical to the problems they had been solving for ML systems since 2020. Phoenix was extracted as an open-source standalone tool in 2023, and the OpenTelemetry-native instrumentation layer was added to make it composable with the existing LLM tracing ecosystem.

The design philosophy is deliberately local-first: you can run Phoenix with zero cloud dependencies, zero authentication setup, and zero data leaving your infrastructure. For teams with strict data governance requirements, this is not a minor convenience - it is the difference between being able to observe your LLM system at all versus not.

Phoenix Architecture: Four Distinct Layers

Instrumentation layer uses OpenInference - Arize's OpenTelemetry semantic conventions for AI systems. Auto-instrumentation hooks for LangChain, LlamaIndex, Anthropic, OpenAI, and other frameworks capture spans without any changes to your application code.

Transport layer uses standard OTLP - the same protocol used by Jaeger, Tempo, and every other OTel-compatible backend. SimpleSpanProcessor for development, BatchSpanProcessor for production.

Storage layer persists traces and embeddings in SQLite by default. For high-volume production, configure a PostgreSQL-backed Phoenix deployment or the hosted Arize platform. The storage format is fully open - you can query it directly with pandas.

Analysis layer provides three independent capabilities: UMAP visualization of embedding distributions, drift computation relative to a reference snapshot, and the LLM evaluation pipeline. These three are the core differentiators that separate Phoenix from trace-only tools.

Core Features: Traces, Embeddings, Evaluations, Datasets

Traces

Phoenix captures the full span tree for every interaction - the same information available in LangSmith and Langfuse. For a RAG pipeline, this means a parent span for the full request, child spans for embedding computation, retrieval, and LLM generation, with token counts, latency, and model metadata on each span. The trace view in Phoenix is adequate for individual-interaction debugging, though LangSmith's trace UI is more polished for this specific use case.

Embedding Visualization (UMAP)

This is Phoenix's primary differentiator. Embedding vectors - whether from query text, document chunks, or model responses - are high-dimensional (typically 768 to 3072 dimensions). UMAP (Uniform Manifold Approximation and Projection) reduces them to 2 or 3 dimensions while preserving neighborhood structure: vectors that were semantically similar in high-dimensional space remain close together in the projection. The result is a navigable map of your system's interactions where geographic proximity means semantic similarity.

LLM Evaluations

Phoenix includes a built-in evaluation library with pre-built evaluators for hallucination detection, answer correctness, and retrieval relevance. Evaluations run as a post-processing step - you collect traces, then judge them with a separate LLM. The evaluation results are linked back to the original traces and visible in the same UMAP view, letting you color-code the embedding space by quality score and immediately identify which semantic regions have quality problems.

Datasets

Phoenix can export any filtered set of traces as a dataset - a structured collection of (input, output, context) triples. These datasets serve as the foundation for regression testing: capture 200 representative traces today, save as a dataset, and use it as your golden set for evaluating prompt changes or model upgrades next month.

Setup and Auto-Instrumentation

The fastest path to Phoenix instrumentation uses auto-instrumentation, which hooks into supported frameworks transparently:

# requirements:
# pip install arize-phoenix openinference-instrumentation-anthropic
# pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

import anthropic
import phoenix as px
from openinference.instrumentation.anthropic import AnthropicInstrumentor
from opentelemetry import trace as otel_trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter


def setup_phoenix_local() -> str:
    """
    Start Phoenix locally and configure OTel instrumentation.

    Returns the Phoenix UI URL for easy access.

    Phoenix launches as a background thread serving:
    - UI at http://localhost:6006
    - OTLP HTTP endpoint at http://localhost:6006/v1/traces
    - OTLP gRPC endpoint at http://localhost:4317 (if configured)
    """
    # Launch the Phoenix server (background thread, non-blocking)
    session = px.launch_app()

    # Configure the OTel TracerProvider to export to Phoenix
    provider = TracerProvider()
    exporter = OTLPSpanExporter(endpoint="http://localhost:6006/v1/traces")

    # SimpleSpanProcessor: synchronous, per-span export - correct for development
    # Use BatchSpanProcessor in production (covered below)
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    otel_trace.set_tracer_provider(provider)

    # Hook into the Anthropic Python SDK
    # Every client.messages.create() call now generates a Phoenix span
    # capturing: messages input, response text, token counts, latency, model name
    AnthropicInstrumentor().instrument()

    print(f"Phoenix UI: {session.url}")
    return session.url


# After setup_phoenix_local(), your application code is completely unchanged.
# The instrumentation is transparent - no try/except, no context managers,
# no manual span creation required for basic tracing.

client = anthropic.Anthropic()


def answer_with_context(question: str, context_documents: list[str]) -> dict:
    """
    Standard RAG answer function - fully traced by Phoenix automatically.

    Phoenix captures without any modification to this function:
    - The full messages array (input)
    - The model's text response (output)
    - Token usage: prompt_tokens, completion_tokens, total_tokens
    - Wall-clock latency from request start to response receipt
    - Model name and version
    - Any exceptions with full tracebacks

    Open http://localhost:6006 after the first call to see the trace.
    """
    context = "\n\n---\n\n".join(
        f"Document {i + 1}:\n{doc}"
        for i, doc in enumerate(context_documents)
    )

    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=1024,
        system=(
            "You are a precise documentation assistant. "
            "Answer questions using only the provided context documents. "
            "If the context does not contain sufficient information to answer, "
            "say explicitly: 'The provided documentation does not cover this topic.'"
        ),
        messages=[
            {
                "role": "user",
                "content": f"Context Documents:\n{context}\n\nQuestion: {question}"
            }
        ]
    )

    return {
        "question": question,
        "answer": response.content[0].text,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "model": response.model,
    }


if __name__ == "__main__":
    url = setup_phoenix_local()

    # Run a few queries - each one produces a trace in Phoenix
    questions_and_docs = [
        (
            "How do I configure webhook retry backoff?",
            [
                "Webhook retry configuration uses the retry_config parameter. "
                "Supported backoff strategies: exponential, linear, fixed.",
                "Exponential backoff doubles the delay between each retry attempt. "
                "Configure with retry_config={'strategy': 'exponential', 'base_delay': 1.0}.",
            ]
        ),
        (
            "What is the rate limit for the search endpoint?",
            [
                "The search API endpoint supports up to 100 requests per minute per API key.",
                "Rate limit headers: X-RateLimit-Remaining, X-RateLimit-Reset.",
            ]
        ),
    ]

    for question, docs in questions_and_docs:
        result = answer_with_context(question, docs)
        print(f"Q: {question}")
        print(f"A: {result['answer'][:150]}...")
        print(f"Tokens: {result['input_tokens']} in / {result['output_tokens']} out\n")

    print(f"Open Phoenix at: {url}")

:::tip Auto-instrumentation is zero-code-change for your application The AnthropicInstrumentor hooks at the HTTP client transport level, not at the function call level. Your application code - every client.messages.create() call - is completely unchanged. The span is created, populated, and exported to Phoenix asynchronously. Typical overhead is under 1 ms per request in production with BatchSpanProcessor configured correctly. :::

Manual Span Creation for RAG Pipeline Steps

Auto-instrumentation captures the LLM call but not the retrieval logic, reranking, or preprocessing steps. For full RAG pipeline visibility, add explicit spans to each step:

import anthropic
import time
import numpy as np
import phoenix as px
from openinference.semconv.trace import SpanAttributes
from opentelemetry import trace as otel_trace
from opentelemetry.trace import Status, StatusCode

tracer = otel_trace.get_tracer("rag-pipeline")
client = anthropic.Anthropic()


def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
    a, b = np.array(vec_a), np.array(vec_b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9))


def get_embedding(text: str) -> list[float]:
    """
    Stub: replace with your actual embedding provider call.
    Returns a normalized 1536-dim embedding vector.
    """
    # In production: call Voyage AI, OpenAI, or Cohere embedding API
    # voyage_client.embed([text], model="voyage-3").embeddings[0]
    rng = np.random.default_rng(abs(hash(text)) % (2**32))
    vec = rng.standard_normal(1536).astype(np.float32)
    return (vec / np.linalg.norm(vec)).tolist()


def rag_pipeline_with_manual_spans(
    query: str,
    document_corpus: list[dict],  # [{"id": str, "text": str, "embedding": list[float]}]
    top_k: int = 3,
) -> dict:
    """
    Full RAG pipeline with manually instrumented spans for each stage.

    Phoenix receives one parent span (rag_request) with three child spans:
    1. embed_query       - query vectorization
    2. retrieve_docs     - cosine similarity retrieval
    3. generate_answer   - LLM generation (also auto-instrumented by AnthropicInstrumentor)

    Attaching embedding vectors to the embed_query span lets Phoenix include
    this query in the UMAP embedding visualization alongside all other queries.
    """
    with tracer.start_as_current_span("rag_request") as root_span:
        root_span.set_attribute("query", query)
        root_span.set_attribute("corpus_size", len(document_corpus))
        root_span.set_attribute("top_k", top_k)
        start_total = time.monotonic()

        # --- Stage 1: Embed the query ---
        with tracer.start_as_current_span("embed_query") as embed_span:
            embed_start = time.monotonic()
            query_embedding = get_embedding(query)
            embed_latency = time.monotonic() - embed_start

            embed_span.set_attribute("embedding.model", "voyage-3")
            embed_span.set_attribute("embedding.dimensions", len(query_embedding))
            embed_span.set_attribute("embed_latency_ms", round(embed_latency * 1000, 2))

            # This attribute is what Phoenix uses to include this query
            # in the UMAP embedding visualization
            embed_span.set_attribute(
                SpanAttributes.EMBEDDING_EMBEDDINGS,
                str([{"vector": query_embedding[:50], "text": query}])  # truncated for span size
            )
            embed_span.set_status(Status(StatusCode.OK))

        # --- Stage 2: Retrieve top-k documents ---
        with tracer.start_as_current_span("retrieve_docs") as retrieval_span:
            retrieve_start = time.monotonic()

            scored_docs = [
                {
                    "id": doc["id"],
                    "text": doc["text"],
                    "score": cosine_similarity(query_embedding, doc["embedding"]),
                }
                for doc in document_corpus
            ]
            top_docs = sorted(scored_docs, key=lambda d: d["score"], reverse=True)[:top_k]

            retrieve_latency = time.monotonic() - retrieve_start

            retrieval_span.set_attribute("retrieval.top_score", round(top_docs[0]["score"], 4))
            retrieval_span.set_attribute("retrieval.mean_score", round(
                sum(d["score"] for d in top_docs) / len(top_docs), 4
            ))
            retrieval_span.set_attribute("retrieval.doc_ids", str([d["id"] for d in top_docs]))
            retrieval_span.set_attribute("retrieve_latency_ms", round(retrieve_latency * 1000, 2))

            # Flag low-confidence retrieval for downstream analysis
            if top_docs[0]["score"] < 0.5:
                retrieval_span.set_attribute("retrieval.low_confidence", True)
                retrieval_span.set_attribute(
                    "retrieval.warning",
                    "Top score below 0.5 - likely a coverage gap"
                )

            retrieval_span.set_status(Status(StatusCode.OK))

        # --- Stage 3: Generate answer with Claude ---
        with tracer.start_as_current_span("generate_answer") as gen_span:
            context = "\n\n".join(
                f"[Source: {doc['id']} | Relevance: {doc['score']:.3f}]\n{doc['text']}"
                for doc in top_docs
            )

            # AnthropicInstrumentor auto-instruments this call - but Phoenix
            # also attributes it to the generate_answer span via context propagation
            response = client.messages.create(
                model="claude-opus-4-6",
                max_tokens=768,
                system=(
                    "Answer the question using only the provided source documents. "
                    "Cite source IDs when making specific claims. "
                    "If sources are insufficient, say so."
                ),
                messages=[{
                    "role": "user",
                    "content": f"Sources:\n{context}\n\nQuestion: {query}"
                }]
            )

            answer = response.content[0].text
            gen_span.set_attribute("answer_length_chars", len(answer))
            gen_span.set_attribute("input_tokens", response.usage.input_tokens)
            gen_span.set_attribute("output_tokens", response.usage.output_tokens)
            gen_span.set_status(Status(StatusCode.OK))

        total_latency = time.monotonic() - start_total
        root_span.set_attribute("total_latency_ms", round(total_latency * 1000, 2))
        root_span.set_status(Status(StatusCode.OK))

        return {
            "query": query,
            "answer": answer,
            "retrieved_docs": top_docs,
            "top_retrieval_score": top_docs[0]["score"],
            "total_latency_ms": round(total_latency * 1000, 2),
        }

Running LLM Evaluations with Phoenix's Evals Library

Beyond tracing and embedding visualization, Phoenix includes a complete LLM evaluation pipeline. Evaluations run as a post-processing step - collect traces during production, then periodically score them with an LLM judge:

import phoenix as px
from phoenix.evals import (
    HallucinationEvaluator,
    QAEvaluator,
    RelevanceEvaluator,
    run_evals,
)
from phoenix.evals.models import AnthropicModel
import pandas as pd
from datetime import datetime, timedelta


def run_production_evaluations(
    project_name: str = "production-rag",
    lookback_hours: int = 24,
    max_traces: int = 500,
) -> dict:
    """
    Run automated quality evaluations on recent production traces.

    Designed to be called on a schedule (e.g., nightly via cron or CI).
    Evaluates the last N hours of production traffic, up to max_traces.

    Args:
        project_name: Phoenix project to pull traces from
        lookback_hours: How far back to look for traces
        max_traces: Upper bound on traces to evaluate (cost control)

    Returns:
        Summary dict with evaluation scores and DataFrame of results
    """
    # Connect to running Phoenix instance
    phoenix_client = px.Client()

    # Pull spans from the specified project
    print(f"Fetching traces from project: {project_name}")
    spans_df = phoenix_client.get_spans_dataframe(project_name=project_name)

    if spans_df.empty:
        print("No spans found - nothing to evaluate.")
        return {"status": "no_data"}

    print(f"Total spans available: {len(spans_df)}")

    # Filter to LLM spans only - these are the ones with input/output/context
    llm_spans = spans_df[spans_df["span_kind"] == "LLM"].copy()
    print(f"LLM spans: {len(llm_spans)}")

    # Sample for cost control
    if len(llm_spans) > max_traces:
        eval_df = llm_spans.sample(n=max_traces, random_state=42)
        print(f"Sampled to {max_traces} traces for evaluation")
    else:
        eval_df = llm_spans.copy()

    # Use Claude Haiku as judge model - 60x cheaper than Opus,
    # 88-92% agreement on binary classification tasks
    judge_model = AnthropicModel(
        model="claude-haiku-4-5-20251001",
        max_tokens=256,
        temperature=0,  # Deterministic for consistent evaluation
    )

    # Three evaluators covering the core RAG quality dimensions
    evaluators = [
        # HallucinationEvaluator: Does the answer contain claims not in the context?
        # Score: 1.0 = not hallucinated, 0.0 = hallucinated
        HallucinationEvaluator(judge_model),

        # QAEvaluator: Does the answer correctly and completely address the question?
        # Score: 1.0 = correct and complete, 0.0 = incorrect or incomplete
        QAEvaluator(judge_model),

        # RelevanceEvaluator: Is the retrieved context relevant to the query?
        # Score: 1.0 = relevant, 0.0 = not relevant
        # Catches retrieval failures independently of generation quality
        RelevanceEvaluator(judge_model),
    ]

    print(f"\nRunning {len(evaluators)} evaluators on {len(eval_df)} traces...")
    print("Using claude-haiku-4-5-20251001 as judge model")

    # run_evals executes evaluators in parallel across the DataFrame
    eval_results = run_evals(
        dataframe=eval_df,
        evaluators=evaluators,
        provide_explanation=True,  # Include reasoning for each judgment
        concurrency=8,             # Parallel judge calls (respect Anthropic rate limits)
    )

    # Merge evaluation results back into the main DataFrame
    result_df = eval_df.copy()
    for result, evaluator in zip(eval_results, evaluators):
        name = evaluator.__class__.__name__.replace("Evaluator", "").lower()
        result_df = result_df.join(
            result[["score", "label", "explanation"]].rename(columns={
                "score": f"{name}_score",
                "label": f"{name}_label",
                "explanation": f"{name}_explanation",
            })
        )

    # Summary statistics
    summary = {
        "evaluated_at": datetime.utcnow().isoformat(),
        "project": project_name,
        "traces_evaluated": len(result_df),
    }

    print("\n=== Evaluation Summary ===")
    for evaluator in evaluators:
        name = evaluator.__class__.__name__.replace("Evaluator", "").lower()
        score_col = f"{name}_score"
        if score_col in result_df.columns:
            mean = result_df[score_col].mean()
            pct_failing = (result_df[score_col] < 0.5).mean() * 100
            summary[f"{name}_mean_score"] = round(mean, 4)
            summary[f"{name}_failure_rate"] = round(pct_failing, 2)
            print(f"  {name:16s}: {mean:.3f} mean | {pct_failing:.1f}% failing")

    return {"summary": summary, "results": result_df}


# Example: run nightly and log results
if __name__ == "__main__":
    output = run_production_evaluations(
        project_name="docs-assistant-prod",
        lookback_hours=24,
        max_traces=200,
    )
    print("\nSummary:", output.get("summary", {}))

:::warning Use Haiku for evaluations, not Opus Running 1,000 quality evaluations with claude-opus-4-6 costs approximately $15 to$20. The same evaluations with claude-haiku-4-5-20251001 cost approximately $0.25. For binary and categorical quality judgments - grounded or not grounded, relevant or not relevant - Haiku's agreement with Opus is 88 to 92% on most benchmark tasks. Reserve Opus for calibration runs on 100 to 200 traces per domain to measure the agreement rate. If Haiku agrees with Opus more than 85% of the time on your data, Haiku is a reliable production judge. Use Opus only when the domain requires nuanced multi-step reasoning that binary categories cannot capture. :::

Custom Evaluator: Claude Haiku as Domain-Specific Judge

The built-in evaluators cover general quality dimensions. For domain-specific criteria - citation accuracy, regulatory compliance language, technical correctness in a specialized domain - you build a custom evaluator using the same Anthropic SDK:

import anthropic
import json
import pandas as pd
from dataclasses import dataclass
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Optional


@dataclass
class EvalResult:
    trace_id: str
    score: float        # 0.0 to 1.0
    label: str          # Human-readable category
    explanation: str    # Judge's reasoning
    tokens_used: int    # For cost tracking


GROUNDEDNESS_PROMPT = """You are a strict factual grounding evaluator.

Your task: determine whether every factual claim in the ANSWER is explicitly supported
by information in the CONTEXT. You are not evaluating completeness or helpfulness -
only whether the answer introduces facts that are NOT present in the context.

CONTEXT:
{context}

QUESTION:
{question}

ANSWER:
{answer}

Evaluation criteria:
- GROUNDED: Every factual claim in the answer traces directly to the context
- PARTIALLY_GROUNDED: Most claims are supported but 1-2 minor details are not in the context
- NOT_GROUNDED: The answer contains significant claims that are fabricated or inferred beyond the context

Respond with a JSON object only - no other text:
{{
  "label": "GROUNDED" | "PARTIALLY_GROUNDED" | "NOT_GROUNDED",
  "score": 1.0 | 0.5 | 0.0,
  "explanation": "One or two sentences explaining your judgment with specific references"
}}"""


def evaluate_single_trace(
    trace: dict,
    client: anthropic.Anthropic,
    judge_model: str = "claude-haiku-4-5-20251001",
) -> EvalResult:
    """
    Evaluate a single trace for answer groundedness.

    Args:
        trace: dict with keys: trace_id, question, context, answer
        client: Anthropic client (shared across calls for connection reuse)
        judge_model: Model ID for the judge - default Haiku for cost efficiency
    """
    prompt = GROUNDEDNESS_PROMPT.format(
        context=trace.get("context", ""),
        question=trace.get("question", ""),
        answer=trace.get("answer", ""),
    )

    try:
        response = client.messages.create(
            model=judge_model,
            max_tokens=256,
            temperature=0,
            messages=[{"role": "user", "content": prompt}]
        )

        raw = response.content[0].text.strip()
        # Strip markdown code fences if the model wraps in ```json
        if raw.startswith("```"):
            raw = raw.split("```")[1]
            if raw.startswith("json"):
                raw = raw[4:]

        result = json.loads(raw)
        return EvalResult(
            trace_id=trace["trace_id"],
            score=float(result.get("score", 0.0)),
            label=result.get("label", "NOT_GROUNDED"),
            explanation=result.get("explanation", ""),
            tokens_used=response.usage.input_tokens + response.usage.output_tokens,
        )

    except (json.JSONDecodeError, KeyError, Exception) as e:
        return EvalResult(
            trace_id=trace["trace_id"],
            score=0.0,
            label="EVAL_ERROR",
            explanation=f"Evaluation failed: {type(e).__name__}: {str(e)[:100]}",
            tokens_used=0,
        )


def batch_evaluate_groundedness(
    traces: list[dict],
    concurrency: int = 8,
    judge_model: str = "claude-haiku-4-5-20251001",
) -> pd.DataFrame:
    """
    Evaluate groundedness across a batch of traces with parallel execution.

    Args:
        traces: List of trace dicts (trace_id, question, context, answer)
        concurrency: Number of parallel judge calls
        judge_model: Model to use for evaluation

    Returns:
        DataFrame with columns: trace_id, score, label, explanation, tokens_used
    """
    client = anthropic.Anthropic()
    results: list[EvalResult] = []
    total_tokens = 0

    print(f"Evaluating {len(traces)} traces with {concurrency} parallel workers")
    print(f"Judge model: {judge_model}")

    with ThreadPoolExecutor(max_workers=concurrency) as executor:
        futures = {
            executor.submit(evaluate_single_trace, trace, client, judge_model): trace["trace_id"]
            for trace in traces
        }

        for i, future in enumerate(as_completed(futures)):
            result = future.result()
            results.append(result)
            total_tokens += result.tokens_used

            if (i + 1) % 25 == 0:
                grounded = sum(1 for r in results if r.score >= 0.5) / len(results)
                estimated_cost = (total_tokens / 1_000_000) * 0.25  # Haiku pricing ~$0.25/M tokens
                print(
                    f"  {i + 1}/{len(traces)} | "
                    f"Grounded: {grounded:.1%} | "
                    f"Est. cost: ${estimated_cost:.4f}"
                )

    df = pd.DataFrame([
        {
            "trace_id": r.trace_id,
            "groundedness_score": r.score,
            "groundedness_label": r.label,
            "groundedness_explanation": r.explanation,
            "tokens_used": r.tokens_used,
        }
        for r in results
    ])

    print(f"\nGroundedness Results:")
    print(f"  GROUNDED:          {(df['groundedness_label'] == 'GROUNDED').sum()}")
    print(f"  PARTIALLY_GROUNDED:{(df['groundedness_label'] == 'PARTIALLY_GROUNDED').sum()}")
    print(f"  NOT_GROUNDED:      {(df['groundedness_label'] == 'NOT_GROUNDED').sum()}")
    print(f"  EVAL_ERROR:        {(df['groundedness_label'] == 'EVAL_ERROR').sum()}")
    print(f"  Mean score:        {df['groundedness_score'].mean():.4f}")
    print(f"  Total tokens used: {total_tokens:,}")

    return df

Embedding Analysis: Capturing Embeddings for UMAP

UMAP visualization requires embedding vectors alongside traces. This example shows the complete embedding capture pattern for a RAG system, including both query and document embeddings:

import anthropic
import numpy as np
import phoenix as px
from openinference.semconv.trace import SpanAttributes
from opentelemetry import trace as otel_trace
from opentelemetry.trace import Status, StatusCode


tracer = otel_trace.get_tracer("embedding-capture")
client = anthropic.Anthropic()


def get_voyage_embedding(text: str) -> list[float]:
    """
    Get text embedding using Voyage AI (recommended with Claude).

    Voyage AI's voyage-3 model is optimized for retrieval tasks and
    integrates well with Claude's semantic space.
    """
    # In production: import voyageai and call the API
    # import voyageai
    # voyage = voyageai.Client()
    # return voyage.embed([text], model="voyage-3").embeddings[0]

    # Stub for illustration - replace with real embedding call
    rng = np.random.default_rng(abs(hash(text[:64])) % (2**32))
    vec = rng.standard_normal(1024).astype(np.float32)
    return (vec / np.linalg.norm(vec)).tolist()


def rag_with_full_embedding_capture(
    query: str,
    document_corpus: list[dict],
    session_id: str,
    user_segment: str = "unknown",
) -> dict:
    """
    RAG pipeline that captures rich embedding metadata for Phoenix UMAP analysis.

    Key embedding capture points:
    1. Query embedding - where in semantic space is this question?
    2. Retrieved document embeddings - what did we pull?
    3. Answer embedding (optional) - how does the response relate to the query?

    The query embedding is the most critical for UMAP visualization.
    Phoenix plots all query embeddings together - this is how you see clusters.

    Session ID and user segment annotations let you color-code the UMAP
    by user type, time period, or any other business dimension.
    """
    with tracer.start_as_current_span("rag_with_embeddings") as span:
        # Annotate with business dimensions (for filtering in Phoenix UI)
        span.set_attribute("session.id", session_id)
        span.set_attribute("user.segment", user_segment)
        span.set_attribute("query.text", query)

        # --- Capture query embedding ---
        query_embedding = get_voyage_embedding(query)

        # OpenInference EMBEDDING_EMBEDDINGS attribute: Phoenix reads this
        # to include this span's query in the UMAP visualization
        # Format: list of embedding dicts, each with "vector" and "text"
        span.set_attribute(
            SpanAttributes.EMBEDDING_EMBEDDINGS,
            str([{
                "vector": query_embedding,
                "text": query,
                "model": "voyage-3",
            }])
        )

        # --- Retrieve documents and capture their embeddings ---
        doc_embeddings = [
            {
                "id": doc["id"],
                "text": doc["text"],
                "embedding": get_voyage_embedding(doc["text"]),
            }
            for doc in document_corpus
        ]

        # Compute similarities
        similarities = [
            float(np.dot(query_embedding, doc["embedding"]) /
                  (np.linalg.norm(query_embedding) * np.linalg.norm(doc["embedding"]) + 1e-9))
            for doc in doc_embeddings
        ]
        top_indices = sorted(range(len(similarities)), key=lambda i: similarities[i], reverse=True)[:3]
        top_docs = [doc_embeddings[i] for i in top_indices]
        top_scores = [similarities[i] for i in top_indices]

        # Capture retrieved document embeddings for coverage analysis
        # Phoenix uses these to overlay document embeddings on the same UMAP
        # as query embeddings - gaps between query clusters and doc clusters = coverage gaps
        span.set_attribute(
            "retrieval.document_embeddings",
            str([{
                "id": doc["id"],
                "vector": doc["embedding"][:10],  # Truncated for span attribute size limits
                "text": doc["text"][:200],
                "score": score,
            } for doc, score in zip(top_docs, top_scores)])
        )

        span.set_attribute("retrieval.top_score", round(top_scores[0], 4))
        span.set_attribute("retrieval.mean_score", round(np.mean(top_scores), 4))

        # Flag for Phoenix filtering: low-confidence retrieval = likely coverage gap
        is_low_confidence = top_scores[0] < 0.5
        span.set_attribute("retrieval.is_low_confidence", is_low_confidence)

        # --- Generate answer ---
        context = "\n\n".join(
            f"[{doc['id']} | score: {score:.3f}]\n{doc['text']}"
            for doc, score in zip(top_docs, top_scores)
        )

        response = client.messages.create(
            model="claude-opus-4-6",
            max_tokens=768,
            system="Answer questions using only the provided documents. Cite document IDs.",
            messages=[{"role": "user", "content": f"Documents:\n{context}\n\nQuestion: {query}"}]
        )

        answer = response.content[0].text

        # Optional: capture answer embedding for response drift analysis
        # answer_embedding = get_voyage_embedding(answer)
        # span.set_attribute("answer.embedding", str(answer_embedding))

        span.set_status(Status(StatusCode.OK))

        return {
            "query": query,
            "answer": answer,
            "top_retrieval_score": top_scores[0],
            "is_low_confidence": is_low_confidence,
            "session_id": session_id,
        }

:::info Why UMAP and not t-SNE or PCA? UMAP is preferred for embedding visualization because it preserves both local structure (nearby points stay nearby) and global structure (clusters remain meaningfully separated) better than t-SNE at scale. PCA is linear and cannot capture the nonlinear manifold structure of language model embedding spaces. UMAP is also significantly faster than t-SNE for datasets above 10,000 points - important for real-time Phoenix dashboards on large production traces. :::

Production Deployment: BatchSpanProcessor and Remote Phoenix

import os
import anthropic
import phoenix as px
from openinference.instrumentation.anthropic import AnthropicInstrumentor
from opentelemetry import trace as otel_trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter


def configure_phoenix_production(
    phoenix_endpoint: Optional[str] = None,
    project_name: Optional[str] = None,
) -> None:
    """
    Production-grade Phoenix instrumentation configuration.

    Key differences from development setup:

    1. BatchSpanProcessor instead of SimpleSpanProcessor
       SimpleSpanProcessor exports one span at a time, synchronously.
       In production, this adds measurable latency to every LLM call.
       BatchSpanProcessor buffers spans in memory and flushes in batches
       on a background thread - the application never waits for export.

    2. gRPC instead of HTTP for lower transport overhead
       OTLPSpanExporter with gRPC uses persistent connections and binary
       encoding instead of HTTP/JSON. Lower CPU and bandwidth at high volume.

    3. Environment-based configuration
       Never hardcode Phoenix endpoints. Use env vars for portability
       across development, staging, and production environments.

    4. Explicit project namespacing
       Use different project names per environment to keep production
       and staging data separated in the Phoenix UI.
    """
    endpoint = phoenix_endpoint or os.environ.get(
        "PHOENIX_COLLECTOR_ENDPOINT",
        "http://localhost:4317"  # Default gRPC port
    )
    project = project_name or os.environ.get(
        "PHOENIX_PROJECT_NAME",
        "production"
    )

    # Set project name as OTel resource attribute
    os.environ["OTEL_SERVICE_NAME"] = project

    provider = TracerProvider()

    # BatchSpanProcessor parameters tuned for typical LLM workloads:
    # - max_queue_size=512: buffer up to 512 spans before dropping (adjust for your volume)
    # - schedule_delay_millis=5000: flush every 5 seconds
    # - max_export_batch_size=128: send up to 128 spans per batch request
    # - export_timeout_millis=30000: give Phoenix 30s to accept spans before timeout
    exporter = OTLPSpanExporter(
        endpoint=endpoint,
        insecure=True,  # Set to False and configure TLS certs for production
    )
    batch_processor = BatchSpanProcessor(
        span_exporter=exporter,
        max_queue_size=512,
        schedule_delay_millis=5000,
        max_export_batch_size=128,
        export_timeout_millis=30_000,
    )
    provider.add_span_processor(batch_processor)
    otel_trace.set_tracer_provider(provider)

    # Instrument with the configured provider
    AnthropicInstrumentor().instrument(
        tracer_provider=provider,
        skip_dep_check=True,  # Skip version check for faster startup
    )

    print(f"Phoenix production tracing configured:")
    print(f"  Endpoint: {endpoint}")
    print(f"  Project:  {project}")
    print(f"  Processor: BatchSpanProcessor (queue=512, interval=5s)")

Docker Compose Deployment for Phoenix Server

# docker-compose.phoenix.yml
# Deploy with: docker-compose -f docker-compose.phoenix.yml up -d

version: "3.8"

services:
  phoenix:
    image: arizephoenix/phoenix:latest
    container_name: phoenix-server
    ports:
      - "6006:6006"   # Phoenix UI and OTLP HTTP endpoint
      - "4317:4317"   # OTLP gRPC endpoint
      - "4318:4318"   # OTLP HTTP endpoint (alternative)
    environment:
      - PHOENIX_WORKING_DIR=/phoenix_data
      # Optional: enable authentication for production
      # - PHOENIX_SECRET=your-secret-here
      # Optional: PostgreSQL backend for high-volume production
      # - PHOENIX_SQL_DATABASE_URL=postgresql://user:pass@postgres:5432/phoenix
    volumes:
      - phoenix_data:/phoenix_data
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6006/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Optional: PostgreSQL backend for high-volume deployments
  # postgres:
  #   image: postgres:16
  #   environment:
  #     POSTGRES_DB: phoenix
  #     POSTGRES_USER: phoenix
  #     POSTGRES_PASSWORD: phoenix_secret
  #   volumes:
  #     - postgres_data:/var/lib/postgresql/data

volumes:
  phoenix_data:
  # postgres_data:

:::danger Never use SimpleSpanProcessor in production SimpleSpanProcessor exports each span synchronously, inline with your request-handling code. For every LLM API call, you are waiting for the span export to complete before returning the response to the user. At P99 latency, this adds 5 to 50ms per request depending on network conditions to your Phoenix server. In high-traffic production, this is unacceptable. BatchSpanProcessor exports on a background thread with no impact on request latency. There is no legitimate reason to use SimpleSpanProcessor outside of development or debugging contexts. :::

Phoenix vs LangSmith vs Langfuse: Comparison

Capability	Phoenix (Arize)	LangSmith	Langfuse
Primary strength	Embedding analysis, distribution-level debugging	Per-trace inspection, prompt iteration	Cost tracking, team collaboration
UMAP visualization	Native, first-class feature	Not available	Not available
Embedding drift detection	Built-in, statistical comparison	Not built-in	Not built-in
RAG coverage gap detection	Visual via UMAP overlay	Manual analysis only	Manual analysis only
LLM evaluation library	Built-in (Hallucination, QA, Relevance)	Annotation UI + LLM evals	LLM scoring (custom prompts)
Prompt playground	Limited	Excellent, first-class	Good
OpenTelemetry support	Native (OpenInference conventions)	Proprietary protocol	OTLP supported
Self-hostable	Yes - pip install + Docker	No (SaaS only)	Yes - Docker
Data leaves your infra	No (local-first)	Yes (cloud)	No (self-hosted option)
Cost	Free (OSS)	$39/month+	Free tier + $49/month
Best for	Distribution debugging, drift, coverage	Per-trace debugging, prompt lab	Cost monitoring, multi-team
Database backend	SQLite (dev), PostgreSQL (prod)	Proprietary	PostgreSQL
Trace retention	Self-managed	30–90 days (plan dependent)	Self-managed
Vendor lock-in risk	Low (OTel native)	High (proprietary)	Low (OTel supported)

:::tip Use Phoenix and LangSmith together - they are not competing Phoenix answers: "What is happening across all interactions? Are there coverage gaps? Has the distribution shifted?" LangSmith answers: "What happened in this specific failing conversation? Let me replay it with a modified prompt." These are different questions requiring different tools. Most production RAG teams run Phoenix for strategic observability (weekly analysis, drift monitoring, coverage audits) and LangSmith for operational debugging (investigating specific reported failures). The OTLP-native instrumentation means you can forward the same traces to both with a second span processor added to your TracerProvider. :::

Common Debugging Workflows

Workflow 1: Diagnosing a Sudden Satisfaction Score Drop

Step-by-step:

1. Open Phoenix and set the time range to the period when satisfaction dropped.
2. Switch to the Embedding tab. Render UMAP for query embeddings.
3. Color nodes by your quality metric - retrieval score, evaluation score, or user rating.
4. Look for dark clusters (low-quality interactions) that form a coherent geographic region.
5. Select the dark cluster. Phoenix shows you the actual queries and answers in that region.
6. Pattern-match: are these all about the same topic? Same query structure? Same time window?
7. Identify root cause and act: missing docs (reindex), prompt failure (adjust), model degradation (rollback).

Workflow 2: Pre-Launch Retrieval Coverage Audit

Before launching a new RAG system or significantly expanding its scope, use Phoenix to verify that your retrieval index actually covers the expected query space:

1. Generate 200 to 500 synthetic queries representative of expected user questions. Use Claude to generate diverse phrasings across all expected topics.
2. Run all synthetic queries through your RAG pipeline with Phoenix instrumentation enabled.
3. Render UMAP with both query embeddings and document chunk embeddings overlaid on the same projection.
4. Identify query clusters that have no nearby document clusters - these are coverage gaps before a single real user experiences them.
5. For each gap: read the sample queries to understand the topic, add documentation, reindex, and re-run the synthetic queries.
6. Accept launch when all major synthetic query clusters have document coverage with mean retrieval score above 0.6.

Workflow 3: Prompt Regression Detection After Changes

Any prompt change or model upgrade risks degrading quality in ways that are not visible in aggregate metrics but visible in specific semantic clusters:

1. Maintain a "golden set" dataset in Phoenix: 100 to 200 challenging queries with ground-truth quality labels.
2. After any prompt change or model upgrade, run the golden set through both the old and new configuration.
3. In Phoenix, compare evaluation scores between the two runs. The UMAP allows you to overlay both runs in the same embedding space.
4. If any cluster shifts toward lower quality in the new configuration, use LangSmith to drill into the specific traces in that cluster and identify what changed.
5. Promotion gate: the new configuration goes to production only if mean evaluation score is equal or better, and no individual cluster score drops by more than 10 percentage points.

Workflow 4: Investigating a New Unknown Cluster

When Phoenix shows a new cluster appearing in the UMAP that was not there last week:

1. Select the cluster and read 20 sample queries. What is the topic?
2. Check mean retrieval score for the cluster. Below 0.5 means a coverage gap - the system has no relevant documents. Above 0.5 but still poor quality means a model or prompt failure for this query type.
3. Check whether the cluster corresponds to a recent external event: a product launch, a Reddit thread, a competitor comparison.
4. Determine the response: coverage gap requires adding documentation; model failure requires prompt analysis in LangSmith; external event may require a deliberate decision about whether to serve that intent.
5. Track the cluster over 48 hours after intervention. Phoenix's time-series view shows whether the quality for that cluster is recovering.

Interview Q&A

Q: What is Phoenix by Arize and what problem does it solve that LangSmith does not?

Phoenix is an open-source ML observability platform that provides embedding-level analysis of LLM systems - specifically UMAP visualization of embedding distributions, drift detection relative to a reference snapshot, and retrieval coverage gap analysis. It solves a class of problem that trace-level tools fundamentally cannot surface: population-level failure patterns that are invisible at the individual-trace level.

LangSmith is excellent at answering "what happened in this specific interaction?" - you inspect the input, retrieved chunks, intermediate steps, and final output for one conversation. Phoenix answers "what is happening across all interactions?" - you see where every query in the past week lives in semantic space, which clusters have low quality, whether new clusters have appeared, and whether your retrieval index has coverage over the full query distribution.

The tools are complementary. LangSmith for operational debugging of specific failures. Phoenix for strategic observability of the system's behavior as a distribution. Most production teams run both.

Q: Explain UMAP embedding visualization. Why is it the key diagnostic tool in Phoenix?

UMAP (Uniform Manifold Approximation and Projection) is a dimensionality reduction algorithm that projects high-dimensional vectors - typically 768 to 3072 dimensions for modern embedding models - into 2D or 3D space while preserving neighborhood structure. Vectors that were semantically similar in high-dimensional space remain geographically close in the UMAP projection.

For LLM observability, this matters because it converts an abstract high-dimensional space into something you can visually inspect. You can see where user queries cluster - all authentication questions cluster together, all billing questions cluster together, all API questions cluster together. You can overlay document embeddings on the same projection and immediately see whether there are query clusters with no nearby document clusters. Those gaps are not visible from metrics or traces - they only exist as a structural property of the embedding distribution.

UMAP is preferred over t-SNE because it preserves global structure (clusters remain meaningfully separated, not just locally coherent) and scales efficiently to tens of thousands of points without prohibitive compute time.

Q: How does Phoenix's LLM evaluation pipeline work? Walk through the architecture.

Phoenix's evaluation pipeline treats quality assessment as a batch post-processing job rather than inline evaluation. The pipeline has three stages.

First, trace collection: your instrumented application sends spans to Phoenix during normal operation. These accumulate in Phoenix's storage. You call phoenix_client.get_spans_dataframe() to retrieve them as a pandas DataFrame.

Second, judge model invocation: you define evaluators (HallucinationEvaluator, QAEvaluator, RelevanceEvaluator, or custom) and call run_evals() with the DataFrame and evaluator list. For each row, the evaluator formats a judge prompt combining the query, context, and answer, then calls the judge model - typically Claude Haiku for cost efficiency. The model returns a score, label, and explanation.

Third, result integration: evaluation results are returned as DataFrames that you join back to the original trace data. Results appear in the Phoenix UI linked to the original traces, and the quality scores can be used to color-code the UMAP.

For the judge model choice: claude-haiku-4-5-20251001 costs approximately 60x less than Opus and achieves 88 to 92% agreement with Opus on binary classification evaluation tasks. Calibrate by running both models on 100 to 200 traces from your specific domain and measuring agreement. Above 85% agreement, Haiku is a reliable production judge.

Q: What is embedding drift and how does Phoenix detect it?

Embedding drift occurs when the distribution of production query embeddings shifts away from the distribution the system was designed for. For RAG systems specifically, drift is dangerous because the retrieval index was built for one query distribution. If the distribution shifts - new user segments arrive, new product features attract new question types, terminology changes - retrieval quality degrades for the drifted queries without any change to the system itself.

Phoenix detects drift by maintaining a reference embedding distribution (a snapshot of production traffic at a defined baseline point - typically launch day or a post-stabilization period) and computing the statistical distance between the reference and the current distribution daily or on a schedule. The distance metrics are KL divergence (measuring information-theoretic difference between the distributions) and Euclidean distance between distribution centroids.

When drift exceeds a threshold (typically 0.2 to 0.5 on Phoenix's normalized scale), Phoenix flags it for investigation. Drift is not always bad - it can mean users have found a valuable new use case. But it always warrants inspection: does the new query distribution have retrieval coverage? Are the new queries being answered well?

Q: A team is considering replacing their Langfuse setup with Phoenix. What are the key trade-offs to evaluate?

This depends heavily on what the team values most. Phoenix's strengths over Langfuse: UMAP embedding visualization (Langfuse has none), drift detection (built-in statistical comparison vs. manual in Langfuse), RAG coverage gap analysis (native in Phoenix, not available in Langfuse), and a richer built-in evaluation library with pre-built evaluators for hallucination, QA correctness, and retrieval relevance.

Langfuse's strengths over Phoenix: better team collaboration features (comments, annotations, role-based access), more mature cost tracking and budget alerting, a cleaner trace UI for per-interaction debugging, and a more active community around prompt version management. Langfuse also has broader framework support in some areas.

The honest recommendation: these tools solve different problems. If the team's primary pain points are "we don't know why satisfaction dropped" or "we need to audit retrieval coverage before launching," Phoenix is the right choice. If the pain points are "we need to track LLM costs across teams" or "we need engineers to annotate and review specific conversations," Langfuse is stronger. For teams with the bandwidth, running Phoenix alongside LangSmith or Langfuse is not redundant - it genuinely adds coverage for the distribution-level observability that neither trace tool provides.

Q: How do you architect Phoenix for a multi-tenant production system where multiple teams' RAG applications need isolation?

Use Phoenix's project naming feature to create per-team, per-application, or per-environment namespaces. In the OTel configuration, set OTEL_SERVICE_NAME to {team}-{app}-{environment} - Phoenix uses this as the project name. Each project appears as a separate namespace in the Phoenix UI with its own UMAP, drift baselines, and evaluation history.

For multi-tenant isolation at the infrastructure level: deploy a single shared Phoenix server (or a small cluster for high volume) accessible to all teams, but configure each team's application to use a different project name. Teams can only see their own project data in the UI - set up basic auth or SSO if teams should not have access to each other's data.

For the reference dataset used in drift detection: each project needs its own reference. Capture the baseline for each team's application independently, two to four weeks after launch when traffic is stable. Store reference snapshots in Phoenix's dataset feature with a naming convention like {project}-baseline-{date}.

At very high volume (millions of spans per day), configure sampling at the application level: 100% of traces in development, 20 to 30% in production. Phoenix's UMAP is statistically reliable at 20% sampling for datasets above 100,000 traces - the projection stabilizes quickly with sufficient coverage.

Production Engineering Notes

Sampling Strategy for High-Traffic Systems

At high query volume, capturing every trace for UMAP analysis is both expensive in storage and unnecessary. UMAP is a statistical tool - it needs representative coverage, not complete coverage. For most production RAG systems, 10 to 20% head-based sampling provides sufficient statistical power for meaningful drift detection and coverage gap analysis. Use 100% capture during incident investigation, golden-set evaluation runs, and the first two weeks post-launch when you are establishing your baseline.

Implement sampling at the TracerProvider level using OTel's built-in ParentBasedTraceIdRatioBased sampler - do not implement sampling in your application code.

Embedding Storage Cost Management

Embeddings are large. A float32 vector with 1536 dimensions occupies 6KB. At 100,000 queries per day with 20% sampling, that is 20,000 embeddings per day at 6KB each - 120MB per day, 3.6GB per month. Phoenix's SQLite default handles this comfortably up to roughly 50GB, but plan storage capacity before reaching that limit. For volumes above 5 million embeddings, configure the PostgreSQL backend or migrate to Arize's hosted platform.

Reference Dataset Selection and Maintenance

The quality of drift detection depends entirely on the quality of your reference dataset. A reference captured during an atypical period - a promotional event, a bot attack, a product launch announcement - will produce spurious drift alerts for normal traffic. Capture your reference over a minimum two-week window representing steady-state production traffic. Exclude any known unusual periods. Refresh the reference quarterly, or immediately after any intentional major change to the system such as a new embedding model, a significant product launch, or a deliberate expansion to a new user segment.

:::info Phoenix's evaluator prompts are open source - read them Phoenix's built-in evaluators (HallucinationEvaluator, QAEvaluator, RelevanceEvaluator) ship with specific judge prompts that are visible in the source code on GitHub. Before deploying them in production, read the actual prompts. They are reasonable general-purpose defaults but are not designed for domain-specific quality criteria. For medical, legal, financial, or other specialized applications, the generic hallucination prompt may not capture the domain-appropriate definition of a factual error. Always read the evaluator prompts, run calibration against your domain-specific data, and build custom evaluators where the built-in criteria do not match your quality requirements. :::

Summary

Phoenix by Arize occupies a distinct and important position in the LLM observability ecosystem. It is not trying to do what LangSmith does - trace-level debugging, prompt playgrounds, conversation replay. It is solving a different problem: what is happening across the entire population of interactions, and how do you make that visible and navigable?

The core technical insight is that high-dimensional embedding spaces contain the answers to population-level questions - but only if you have a tool that can render them into something human-navigable. UMAP projection is that tool. Phoenix builds the surrounding infrastructure: OpenTelemetry-native instrumentation so embedding capture is as simple as adding a span attribute, drift detection so distribution shift triggers investigation before it becomes a crisis, and an LLM evaluation pipeline so quality scores can be overlaid on the same embedding map.

For any team running a RAG system in production, Phoenix addresses the most dangerous class of failure mode: structural problems that look like random noise at the individual-trace level but are systematic patterns across thousands of interactions. Retrieval coverage gaps, query drift, cluster-level quality degradation - these are invisible to trace tools and measurable only at the distribution level. Phoenix makes the distribution visible.

The practical recommendation: add Phoenix to your observability stack alongside whatever trace-level tool you are already using. Use LangSmith or Langfuse for daily operational debugging. Use Phoenix for weekly distribution reviews, pre-launch coverage audits, and drift monitoring. The two types of observability are complementary, and the combination gives you visibility at every level - from the individual conversation to the full semantic landscape of your system's behavior.