Trajectory Evaluation
Which Agent Is Better?
You deploy two agent versions on the same task set. Both complete the tasks successfully. Agent A uses an average of 8 tool calls per task, moves in a clean logical sequence, and never backtracks. Agent B uses an average of 23 tool calls, tries the wrong approach twice on each task, backtracks to correct itself, and eventually reaches the same answer.
Same final output. Wildly different paths.
Which agent would you deploy to production?
The answer is obvious - Agent A is better. But your current evaluation framework, which only checks final output correctness, gives them identical scores. You would have no principled way to prefer Agent A over Agent B.
This is why trajectory evaluation exists. The path to the answer matters. An inefficient trajectory means more cost, more latency, and - critically - more opportunity for something to go wrong. A 23-step trajectory that reaches the right answer today might fail on the 24th step tomorrow when the environment changes slightly.
:::tip 🎮 Interactive Playground Visualize this concept: Try the Agent Trajectory Evaluation demo on the EngineersOfAI Playground - no code required. :::
What a Trajectory Is
A trajectory is the complete, ordered record of everything an agent does from the moment it receives a task to the moment it produces a final output.
Formally, a trajectory is a sequence of tuples:
Where:
- is the state at step - the current context, conversation history, tool results accumulated so far
- is the action at step - either an LLM response, a tool call, or a final answer
- is the observation at step - the result of the action (tool output, LLM response, etc.)
In practice, a trajectory looks like:
Step 1: LLM decides to search for "transformer architecture papers 2023"
Step 2: web_search("transformer architecture papers 2023") → [list of 5 papers]
Step 3: LLM decides to read the first paper
Step 4: read_paper(url="...") → [abstract and key claims]
Step 5: LLM decides to search for more recent papers
Step 6: web_search("transformer improvements 2024") → [list of 5 papers]
Step 7: LLM synthesizes and produces final answer
Every step is observable. Every step is measurable. The trajectory is not just a path to the answer - it is evidence about how the agent thinks, what it prioritizes, and where it struggles.
Why Trajectory Evaluation Matters
Final output quality is a lagging indicator. A high-quality final output might come from a fragile, expensive trajectory. A poor trajectory predicts:
Higher cost: More tool calls and LLM calls mean more tokens, more API costs. An agent that takes 23 steps to do what Agent A does in 8 costs roughly 3x as much.
Higher latency: Each step adds wall-clock time. Tool calls often involve network I/O. An inefficient trajectory means slower responses.
Higher failure risk: Each step is an opportunity for something to go wrong - tool error, hallucination, context overflow. More steps = more risk.
Poor generalization: An agent that reaches the right answer via an inefficient path is often doing something fragile - relying on specific wording in a search result, for example, rather than genuinely understanding the task. It will fail on variations.
Trajectory evaluation catches all of this before the agent reaches production.
Trajectory Quality Dimensions
Trajectory Metrics in Detail
1. Length Efficiency
Definition: The ratio of the minimum theoretically required steps to the actual steps taken.
Where min_steps is the minimum number of steps a perfect agent would need. For a task requiring one tool call and one final response, min_steps = 2. For a task requiring web search + paper reading + synthesis, min_steps = 5.
A score of 1.0 means the agent took the minimum possible path. A score of 0.3 means the agent used 3x more steps than necessary.
Estimation approaches: For benchmarked tasks, you can pre-compute min_steps via expert annotation. For production tasks, you can estimate it as 2 * (number of tools the task type typically requires) + 1.
2. Tool Precision
Definition: The fraction of tool calls that were appropriate and useful.
A tool call is "relevant" if:
- It is the correct tool for the current subtask
- The tool input is well-formed and appropriate
- The tool output was actually used in the subsequent reasoning
Annotating relevance requires either human judgment or LLM-as-judge. For automatic approximation, count tool calls where the subsequent LLM step explicitly references the tool output.
3. Backtracking Rate
Definition: The fraction of steps that reverse or redo a previous decision.
A backtracking step is any step where the agent:
- Calls a tool it already called with the same or very similar input
- Explicitly states it is taking a different approach
- Abandons a line of reasoning it was pursuing
Lower is better. A backtracking rate above 0.20 suggests the agent lacks a coherent plan.
4. Recovery Quality
Definition: When the agent encounters a tool error or dead end, how many steps does it take to recover and continue productively?
Recovery steps: count the number of steps between when an error is first detected and when the agent produces the next productive action. A 1-step recovery (immediate pivot) scores 1.0. A 5-step recovery (extended confusion) scores poorly.
5. Token Efficiency
Definition: The ratio of useful information extracted to total tokens consumed.
This is the hardest metric to compute automatically. A proxy:
Lower token counts for the same task outcome means higher efficiency. Useful as a relative metric when comparing two agent versions on the same task set.
6. Error Rate
Definition: The fraction of tool calls that produced errors.
A tool error is any tool call that returns an error message, raises an exception, or returns an empty/null result. This is fully automatic - no annotation required. Target error rates below 0.10 for production agents.
Trajectory Annotation Schema
For human-annotated evaluation, use a structured annotation schema that annotators fill for each trajectory:
{
"trajectory_id": "traj_abc123",
"task_id": "task_001",
"overall_quality": 4,
"dimensions": {
"goal_understanding": {
"score": 5,
"note": "Agent immediately identified the core subtasks"
},
"tool_selection": {
"score": 3,
"note": "Used read_file when list_files would have been more efficient first"
},
"reasoning_quality": {
"score": 4,
"note": "Good synthesis but missed one edge case"
},
"efficiency": {
"score": 2,
"note": "Took 15 steps for what needed 6"
},
"error_handling": {
"score": 5,
"note": "Cleanly handled the API timeout in step 8"
}
},
"critical_steps": [3, 8, 12],
"backtrack_steps": [9, 10],
"annotator_id": "annotator_02",
"annotation_time_seconds": 420
}
Automatic Trajectory Scoring
Heuristic Rules
Fast, zero-cost heuristics that run on every trajectory without LLM calls:
def heuristic_trajectory_score(trajectory: list) -> dict:
"""Apply heuristic rules to score a trajectory."""
tool_calls = [s for s in trajectory if s["type"] == "tool_call"]
errors = [s for s in tool_calls if s.get("error")]
# Detect backtracking: same tool + similar input within 5 steps
backtrack_count = 0
for i in range(len(tool_calls)):
for j in range(max(0, i - 5), i):
if (tool_calls[i]["name"] == tool_calls[j]["name"] and
_input_similarity(tool_calls[i]["input"], tool_calls[j]["input"]) > 0.8):
backtrack_count += 1
break
return {
"error_rate": len(errors) / len(tool_calls) if tool_calls else 0.0,
"backtracking_rate": backtrack_count / len(tool_calls) if tool_calls else 0.0,
"step_count": len(trajectory),
"tool_call_count": len(tool_calls),
}
LLM-Based Step Evaluation
For each step in the trajectory, ask an LLM judge to rate whether the action was appropriate:
def llm_step_evaluator(step: dict, context: dict) -> float:
"""Score a single step using LLM judgment. Returns 0.0-1.0."""
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=200,
system="You are an expert evaluator of AI agent behavior. "
"Rate the appropriateness of the following agent action on a scale of 0-10. "
"Respond with only a number.",
messages=[{
"role": "user",
"content": f"Task: {context['task']}\n"
f"Previous steps: {context['previous_steps']}\n"
f"Current action: {step}\n"
f"Rate the appropriateness of this action (0-10):"
}],
)
try:
score = float(response.content[0].text.strip()) / 10.0
return max(0.0, min(1.0, score))
except ValueError:
return 0.5
Step-Level vs Trajectory-Level Evaluation
Both granularities are useful, for different purposes:
| Evaluation Level | When to Use | What It Catches |
|---|---|---|
| Step-level | Debugging specific failure modes | Wrong tool selection, malformed inputs |
| Step-level | Training reward models | Step-by-step signal for RL fine-tuning |
| Trajectory-level | Comparing agent versions | Overall efficiency and quality |
| Trajectory-level | Release decisions | End-to-end performance |
| Both | Root cause analysis | Which steps caused trajectory failures |
For regression detection, trajectory-level is primary. If trajectory quality drops, drill into step-level scores to find the cause.
Full Python: Trajectory Recorder and Scorer
"""
Complete trajectory recording and scoring system.
Records full (state, action, observation) sequences and computes 6 metrics.
"""
import json
import math
import time
import hashlib
from dataclasses import dataclass, field, asdict
from typing import Any, Optional
import anthropic
client = anthropic.Anthropic()
# ── Data models ────────────────────────────────────────────────────────────────
@dataclass
class TrajectoryStep:
step_id: int
step_type: str # "llm_call", "tool_call", "observation", "final"
timestamp: float
duration_ms: float
# State: context at time of action
state_summary: str # abbreviated context
# Action
action_type: str # "text_response", "tool_call", "final_answer"
action_content: Any # text, tool name+args, final text
# Observation
observation: Optional[str] = None # tool output or None
# Metadata
input_tokens: int = 0
output_tokens: int = 0
tool_name: Optional[str] = None
tool_input: Optional[dict] = None
error: Optional[str] = None
is_backtrack: bool = False
def to_dict(self) -> dict:
return asdict(self)
@dataclass
class Trajectory:
trajectory_id: str
task_id: str
task_description: str
steps: list[TrajectoryStep] = field(default_factory=list)
final_output: Optional[str] = None
completed: bool = False
start_time: float = field(default_factory=time.time)
end_time: Optional[float] = None
def add_step(self, step: TrajectoryStep):
self.steps.append(step)
def complete(self, final_output: str):
self.final_output = final_output
self.completed = True
self.end_time = time.time()
@property
def total_steps(self) -> int:
return len(self.steps)
@property
def tool_calls(self) -> list[TrajectoryStep]:
return [s for s in self.steps if s.step_type == "tool_call"]
@property
def llm_calls(self) -> list[TrajectoryStep]:
return [s for s in self.steps if s.step_type == "llm_call"]
@property
def total_input_tokens(self) -> int:
return sum(s.input_tokens for s in self.steps)
@property
def total_output_tokens(self) -> int:
return sum(s.output_tokens for s in self.steps)
@property
def total_duration_ms(self) -> float:
return sum(s.duration_ms for s in self.steps)
@dataclass
class TrajectoryScores:
trajectory_id: str
task_id: str
# Six core metrics (all in [0, 1], higher = better)
length_efficiency: float # min_steps / actual_steps
tool_precision: float # correct tool calls / total tool calls
backtracking_rate: float # 1 - (backtrack_steps / total_steps)
recovery_quality: float # quality of error recovery
token_efficiency: float # information per token
error_rate_score: float # 1 - (errors / tool_calls)
# Composite
composite_score: float # weighted average
# Raw counts for debugging
total_steps: int
total_tool_calls: int
error_count: int
backtrack_count: int
total_tokens: int
def to_dict(self) -> dict:
return asdict(self)
# ── Trajectory recorder ────────────────────────────────────────────────────────
class TrajectoryRecorder:
"""
Records a full agent trajectory during execution.
Wraps your agent loop to capture every action and observation.
"""
def __init__(self, task_id: str, task_description: str):
import uuid
self.trajectory = Trajectory(
trajectory_id=str(uuid.uuid4())[:12],
task_id=task_id,
task_description=task_description,
)
self._step_counter = 0
def record_llm_call(
self,
state_summary: str,
llm_response: str,
input_tokens: int,
output_tokens: int,
duration_ms: float,
) -> TrajectoryStep:
self._step_counter += 1
step = TrajectoryStep(
step_id=self._step_counter,
step_type="llm_call",
timestamp=time.time(),
duration_ms=duration_ms,
state_summary=state_summary,
action_type="text_response",
action_content=llm_response,
input_tokens=input_tokens,
output_tokens=output_tokens,
)
self.trajectory.add_step(step)
return step
def record_tool_call(
self,
state_summary: str,
tool_name: str,
tool_input: dict,
tool_output: Optional[str],
duration_ms: float,
error: Optional[str] = None,
) -> TrajectoryStep:
self._step_counter += 1
# Detect backtracking: same tool + similar input recently
is_backtrack = self._detect_backtrack(tool_name, tool_input)
step = TrajectoryStep(
step_id=self._step_counter,
step_type="tool_call",
timestamp=time.time(),
duration_ms=duration_ms,
state_summary=state_summary,
action_type="tool_call",
action_content={"tool": tool_name, "input": tool_input},
observation=tool_output,
tool_name=tool_name,
tool_input=tool_input,
error=error,
is_backtrack=is_backtrack,
)
self.trajectory.add_step(step)
return step
def record_final_answer(self, answer: str) -> TrajectoryStep:
self._step_counter += 1
step = TrajectoryStep(
step_id=self._step_counter,
step_type="final",
timestamp=time.time(),
duration_ms=0,
state_summary="final",
action_type="final_answer",
action_content=answer,
)
self.trajectory.add_step(step)
self.trajectory.complete(answer)
return step
def _detect_backtrack(self, tool_name: str, tool_input: dict) -> bool:
"""Check if this tool call repeats a recent one."""
recent_tool_calls = [
s for s in self.trajectory.steps[-8:]
if s.step_type == "tool_call"
]
input_hash = hashlib.md5(
json.dumps(tool_input, sort_keys=True).encode()
).hexdigest()[:8]
for prev in recent_tool_calls:
if prev.tool_name == tool_name:
prev_hash = hashlib.md5(
json.dumps(prev.tool_input or {}, sort_keys=True).encode()
).hexdigest()[:8]
if input_hash == prev_hash:
return True
return False
# ── Trajectory scorer ──────────────────────────────────────────────────────────
class TrajectoryScorer:
"""
Computes all 6 trajectory quality metrics.
"""
def __init__(self, min_steps_oracle: Optional[dict] = None):
"""
min_steps_oracle: dict mapping task_id -> estimated minimum steps.
If not provided, uses a heuristic based on tool call count.
"""
self.min_steps_oracle = min_steps_oracle or {}
def score(self, trajectory: Trajectory) -> TrajectoryScores:
steps = trajectory.steps
tool_calls = trajectory.tool_calls
n_steps = len(steps)
n_tool_calls = len(tool_calls)
error_count = sum(1 for s in tool_calls if s.error is not None)
backtrack_count = sum(1 for s in tool_calls if s.is_backtrack)
# 1. Length efficiency
min_steps = self.min_steps_oracle.get(trajectory.task_id)
if min_steps is None:
# Heuristic: 2 * unique tool types + 1 LLM synthesis
unique_tools = len({s.tool_name for s in tool_calls if s.tool_name})
min_steps = max(2, unique_tools * 2 + 1)
length_efficiency = min(1.0, min_steps / max(1, n_steps))
# 2. Tool precision
# Approximate: tool calls without errors and not backtracks are "correct"
if n_tool_calls > 0:
correct_calls = n_tool_calls - error_count - backtrack_count
tool_precision = max(0.0, correct_calls / n_tool_calls)
else:
tool_precision = 1.0 # No tool calls needed
# 3. Backtracking rate (converted to score: lower backtracking = higher score)
if n_steps > 0:
backtracking_score = 1.0 - (backtrack_count / n_steps)
else:
backtracking_score = 1.0
# 4. Recovery quality
recovery_quality = self._compute_recovery_quality(steps)
# 5. Token efficiency
total_tokens = trajectory.total_input_tokens + trajectory.total_output_tokens
useful_steps = n_tool_calls - error_count # Non-error tool calls as proxy for useful
if total_tokens > 0 and useful_steps > 0:
# Normalize: more information per token = better
# Perfect: 500 tokens per useful step
# Poor: 5000+ tokens per useful step
tokens_per_useful_step = total_tokens / max(1, useful_steps)
token_efficiency = min(1.0, 500 / max(500, tokens_per_useful_step))
else:
token_efficiency = 0.5 # Unknown
# 6. Error rate score
if n_tool_calls > 0:
error_rate_score = 1.0 - (error_count / n_tool_calls)
else:
error_rate_score = 1.0
# Composite: weighted average
weights = {
"length_efficiency": 0.20,
"tool_precision": 0.25,
"backtracking_rate": 0.15,
"recovery_quality": 0.15,
"token_efficiency": 0.10,
"error_rate_score": 0.15,
}
# Bonus if task completed
completion_bonus = 0.10 if trajectory.completed else 0.0
weights_sum = sum(weights.values())
composite = (
weights["length_efficiency"] * length_efficiency +
weights["tool_precision"] * tool_precision +
weights["backtracking_rate"] * backtracking_score +
weights["recovery_quality"] * recovery_quality +
weights["token_efficiency"] * token_efficiency +
weights["error_rate_score"] * error_rate_score
) / weights_sum
composite = min(1.0, composite + completion_bonus)
return TrajectoryScores(
trajectory_id=trajectory.trajectory_id,
task_id=trajectory.task_id,
length_efficiency=round(length_efficiency, 3),
tool_precision=round(tool_precision, 3),
backtracking_rate=round(backtracking_score, 3),
recovery_quality=round(recovery_quality, 3),
token_efficiency=round(token_efficiency, 3),
error_rate_score=round(error_rate_score, 3),
composite_score=round(composite, 3),
total_steps=n_steps,
total_tool_calls=n_tool_calls,
error_count=error_count,
backtrack_count=backtrack_count,
total_tokens=total_tokens,
)
def _compute_recovery_quality(self, steps: list[TrajectoryStep]) -> float:
"""
After each tool error, count steps until next successful tool call.
Recovery quality = 1 - avg(recovery_steps - 1) / max_recovery_steps
"""
MAX_RECOVERY = 5
recovery_lengths = []
i = 0
while i < len(steps):
step = steps[i]
if step.step_type == "tool_call" and step.error:
# Count steps until next successful tool call
recovery_steps = 0
j = i + 1
while j < len(steps) and recovery_steps < MAX_RECOVERY:
recovery_steps += 1
if (steps[j].step_type == "tool_call" and
not steps[j].error):
break
j += 1
recovery_lengths.append(recovery_steps)
i += 1
if not recovery_lengths:
return 1.0 # No errors = perfect recovery
avg_recovery = sum(recovery_lengths) / len(recovery_lengths)
return max(0.0, 1.0 - (avg_recovery - 1) / MAX_RECOVERY)
# ── Version comparison ─────────────────────────────────────────────────────────
class TrajectoryComparator:
"""
Compare trajectory scores across two agent versions.
Detects regressions and improvements.
"""
def __init__(self, scorer: TrajectoryScorer):
self.scorer = scorer
def compare_versions(
self,
baseline_trajectories: list[Trajectory],
candidate_trajectories: list[Trajectory],
) -> dict:
"""
Compare baseline vs candidate on matched tasks.
Returns regression report.
"""
# Index by task_id
baseline_by_task = {t.task_id: t for t in baseline_trajectories}
candidate_by_task = {t.task_id: t for t in candidate_trajectories}
shared_tasks = set(baseline_by_task) & set(candidate_by_task)
if not shared_tasks:
return {"error": "No shared tasks between versions"}
metrics = [
"length_efficiency", "tool_precision", "backtracking_rate",
"recovery_quality", "token_efficiency", "error_rate_score",
"composite_score"
]
results = {
"shared_tasks": len(shared_tasks),
"metrics": {},
}
for metric in metrics:
baseline_scores = []
candidate_scores = []
for task_id in shared_tasks:
b_scores = self.scorer.score(baseline_by_task[task_id])
c_scores = self.scorer.score(candidate_by_task[task_id])
baseline_scores.append(getattr(b_scores, metric))
candidate_scores.append(getattr(c_scores, metric))
b_mean = sum(baseline_scores) / len(baseline_scores)
c_mean = sum(candidate_scores) / len(candidate_scores)
delta = c_mean - b_mean
delta_pct = (delta / b_mean * 100) if b_mean > 0 else 0
results["metrics"][metric] = {
"baseline_mean": round(b_mean, 3),
"candidate_mean": round(c_mean, 3),
"delta": round(delta, 3),
"delta_pct": round(delta_pct, 1),
"regression": delta < -0.05, # > 5% drop is a regression
"improvement": delta > 0.05, # > 5% gain is an improvement
}
# Summary
regressions = [m for m, v in results["metrics"].items() if v["regression"]]
improvements = [m for m, v in results["metrics"].items() if v["improvement"]]
results["summary"] = {
"regressions": regressions,
"improvements": improvements,
"pass": len(regressions) == 0,
}
return results
def print_comparison_report(self, comparison: dict):
print("\n── Trajectory Comparison Report ────────────────")
print(f"Shared tasks: {comparison['shared_tasks']}")
print(f"\n{'Metric':<25} {'Baseline':>10} {'Candidate':>10} {'Delta':>8} {'Status':>12}")
print("─" * 70)
for metric, values in comparison["metrics"].items():
status = "REGRESSION" if values["regression"] else ("IMPROVED" if values["improvement"] else "unchanged")
print(
f"{metric:<25} {values['baseline_mean']:>10.3f} "
f"{values['candidate_mean']:>10.3f} "
f"{values['delta_pct']:>+7.1f}% {status:>12}"
)
summary = comparison["summary"]
print(f"\nResult: {'PASS' if summary['pass'] else 'FAIL'}")
if summary["regressions"]:
print(f"Regressions: {', '.join(summary['regressions'])}")
if summary["improvements"]:
print(f"Improvements: {', '.join(summary['improvements'])}")
# ── Demo agent using the recorder ─────────────────────────────────────────────
def run_agent_with_recording(task_id: str, task: str) -> Trajectory:
"""Run a simple agent, recording the full trajectory."""
recorder = TrajectoryRecorder(task_id=task_id, task_description=task)
tools = [
{
"name": "web_search",
"description": "Search the web for information.",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string", "description": "Search query"}},
"required": ["query"],
},
},
{
"name": "calculate",
"description": "Perform a calculation.",
"input_schema": {
"type": "object",
"properties": {"expression": {"type": "string", "description": "Math expression"}},
"required": ["expression"],
},
},
]
messages = [{"role": "user", "content": task}]
system = "You are a helpful assistant. Use tools when needed. Be efficient."
for _ in range(15): # max 15 steps
t0 = time.time()
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=2048,
system=system,
tools=tools,
messages=messages,
)
duration_ms = (time.time() - t0) * 1000
# Extract text
text = next((b.text for b in response.content if hasattr(b, "text")), "")
recorder.record_llm_call(
state_summary=f"Messages: {len(messages)}",
llm_response=text,
input_tokens=response.usage.input_tokens,
output_tokens=response.usage.output_tokens,
duration_ms=duration_ms,
)
if response.stop_reason == "end_turn":
recorder.record_final_answer(text)
break
if response.stop_reason == "tool_use":
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type == "tool_use":
t0 = time.time()
# Mock tool execution
if block.name == "web_search":
result = f"Search results for: {block.input.get('query', '')}"
error = None
elif block.name == "calculate":
try:
result = str(eval(block.input.get("expression", "0"))) # noqa: S307
error = None
except Exception as e:
result = None
error = str(e)
else:
result = "Unknown tool"
error = None
tool_duration = (time.time() - t0) * 1000
recorder.record_tool_call(
state_summary=f"Tool: {block.name}",
tool_name=block.name,
tool_input=block.input,
tool_output=result,
duration_ms=tool_duration,
error=error,
)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result if result else f"Error: {error}",
})
messages.append({"role": "user", "content": tool_results})
return recorder.trajectory
# ── Main demo ──────────────────────────────────────────────────────────────────
def main():
scorer = TrajectoryScorer(
min_steps_oracle={
"task_001": 3, # 1 tool call + 1 LLM + 1 final = 3
"task_002": 5,
}
)
comparator = TrajectoryComparator(scorer)
print("Recording trajectories for baseline agent...")
baseline_trajectories = [
run_agent_with_recording("task_001", "What is 15 * 23 + 47?"),
run_agent_with_recording("task_002", "What is the capital of Germany?"),
]
for traj in baseline_trajectories:
scores = scorer.score(traj)
print(f"\nTask {traj.task_id}:")
print(f" Steps: {scores.total_steps} | Tool calls: {scores.total_tool_calls}")
print(f" Length efficiency: {scores.length_efficiency:.3f}")
print(f" Tool precision: {scores.tool_precision:.3f}")
print(f" Backtracking: {scores.backtracking_rate:.3f}")
print(f" Error rate score: {scores.error_rate_score:.3f}")
print(f" Composite score: {scores.composite_score:.3f}")
if __name__ == "__main__":
main()
Production Engineering Notes
Store Trajectories, Not Just Scores
Raw trajectory data is invaluable. Store every step - tool inputs, outputs, token counts, timestamps - in a database (PostgreSQL JSONB column works well). Scores computed today can be recomputed with new metrics tomorrow. Raw trajectories cannot be regenerated.
Use Hashing to Detect Prompt Regressions
Hash your system prompt and include it as metadata on every trajectory. When scores drop, correlate with prompt hash changes to quickly identify whether a prompt edit caused the regression.
Percentile Metrics, Not Just Means
Mean composite score hides variance. An agent with a mean of 0.75 and standard deviation of 0.30 is very different from one with mean 0.72 and std dev 0.05. Report p25, p50, p75, p95 for all trajectory metrics.
:::danger Do Not Use Final Output Alone Evaluating only the final output and ignoring the trajectory misses 60-80% of quality signal. An agent that reaches a correct answer via a 30-step disaster is not a good agent - it is a fragile one that will fail when the environment changes slightly. Always evaluate trajectory quality alongside final output quality. :::
:::warning Backtracking Detection Requires Tuning The simple hash-based backtracking detector shown above will produce false positives when the same tool is called legitimately multiple times with similar inputs (e.g., paginated search). Tune your similarity threshold and lookback window for your specific agent and tool set. :::
Interview Q&A
Q: What is a trajectory in agent evaluation, and why is it important to evaluate?
A: A trajectory is the complete ordered sequence of (state, action, observation) tuples that an agent produces while working on a task - every LLM call, every tool call, every result, from start to final output. Evaluating trajectories is important because the path to the answer matters as much as the answer itself. A trajectory that uses 3x more steps than necessary costs 3x more, takes 3x longer, and has 3x more opportunities for something to go wrong. Final output evaluation alone is a lagging indicator - it misses cost, latency, robustness, and failure mode information that trajectories reveal directly.
Q: Define length efficiency and explain when it would be misleading.
A: Length efficiency is the ratio of the minimum theoretically required steps to the actual steps taken: min_steps / actual_steps. It would be misleading when the minimum steps estimate is wrong. If the task actually requires 10 steps but we estimate 5 as the minimum, every agent looks inefficient. It also misleads when tasks require adaptive behavior - sometimes exploring an unexpected path takes more steps but is the correct response to an uncertain environment. The metric should be interpreted alongside other signals, not in isolation.
Q: How would you compare trajectory quality across two versions of an agent?
A: Run both versions on the same set of evaluation tasks. For each task, compute trajectory scores (length efficiency, tool precision, backtracking rate, recovery quality, token efficiency, error rate) for both versions. Compute mean scores per metric across all tasks. Calculate delta and percentage change for each metric. Flag any metric where the candidate version shows a drop of more than 5% as a regression. Require all regressions to be resolved before deploying the candidate. Also review the distribution, not just the mean - a candidate that improves mean score but increases variance may be worse in practice.
Q: What is the backtracking rate metric, and what does a high value indicate?
A: Backtracking rate is the fraction of steps where the agent undoes or repeats a previous decision - calling the same tool with the same input, explicitly abandoning an approach it was pursuing, or otherwise moving backward in the task. A high backtracking rate (above 20%) indicates the agent lacks a coherent plan. It is trying approaches, discovering they do not work, and trying again - a pattern that looks like intelligent persistence but actually signals poor initial planning or weak task decomposition. In production, high backtracking means high cost and high latency for the same outcome.
Q: How do you handle the "multiple valid trajectories" problem when scoring trajectory quality?
A: The multiple valid trajectories problem means there is no single "correct" trajectory to compare against. The solution is to evaluate properties of the trajectory rather than the trajectory itself. Rather than asking "is this the right sequence of steps?", ask "does this trajectory have the properties of a good trajectory?" - low error rate, no unnecessary backtracking, efficient tool use, good error recovery. These properties are invariant across different valid paths to the same goal. An agent that reaches the answer in 5 clean steps using a different tool sequence than the "expected" one should score just as well as one that uses the expected sequence.
Q: Why should trajectory evaluation be part of a CI/CD pipeline for agents?
A: Including trajectory evaluation in CI/CD catches behavioral regressions before they reach production - the same reason unit tests belong in CI. Without it, a developer changes the system prompt Tuesday night, quality degrades Wednesday morning, and nobody notices until a user complains Thursday. With trajectory evaluation in CI: the pipeline runs 50-100 representative tasks against the proposed change, computes trajectory scores, and blocks the merge if any score drops more than 5% below baseline. This transforms trajectory evaluation from a periodic diagnostic tool into a continuous quality gate - exactly where it belongs in a production engineering workflow.
Q: How do you choose the set of trajectory metrics to track for a specific agent?
A: Start with the universal metrics - step count, tool precision, backtracking rate, error rate - because these apply to almost any agentic system. Then add domain-specific metrics based on your agent's task type. A coding agent should track test pass rate per patch attempt. A research agent should track source citation rate (does it actually cite sources for factual claims?). A customer support agent should track escalation rate (does it know when to hand off to a human?). The selection criterion: only track a metric if you would actually change the agent's behavior when the metric regresses. Metrics you do not act on are noise. Start with three to five metrics and add more as you identify the specific failure modes that matter most for your deployment.
Summary: The Trajectory Evaluation Stack
Trajectory evaluation is not one thing - it is a layered stack of complementary signals:
Each layer answers a different question:
- Step-level: which specific tool calls fail or are wasteful?
- Trajectory-level: is the overall path efficient and well-planned?
- Outcome: did the agent accomplish its goal?
A complete trajectory evaluation system measures all three, stores raw data for reanalysis, and surfaces actionable insights rather than just numbers. The investment pays off quickly: teams that track trajectories catch regressions that would take weeks to surface through user complaints, and they have the data to diagnose exactly why performance changed.
:::tip Key Takeaway Trajectory evaluation is the bridge between "did the agent succeed?" and "how well did the agent work?" Final answer correctness tells you the first. Trajectory metrics tell you the second. Both are necessary for production-quality agents. An agent that scores 90% on final answer correctness but uses 4x more steps than necessary on every task is not production-ready - it is expensive, slow, and fragile. Trajectory evaluation is what reveals this before users do. :::
Further Reading
- Module 08 Lesson 01: Challenges of Evaluating Agents - why trajectory evaluation exists and what problem it solves
- Module 08 Lesson 05: LLM-as-Agent-Judge - using LLMs to evaluate trajectory quality when programmatic scoring is not sufficient
- Module 08 Lesson 07: Production Agent Monitoring - applying trajectory metrics in real-time production observability
- Yao et al. (2022), "ReAct: Synergizing Reasoning and Acting in Language Models" - the foundational paper that defines the (state, action, observation) trajectory formalism used throughout this lesson
- Module 08 Lesson 03: GAIA Benchmark - see how trajectory quality correlates with benchmark performance on real-world tasks