CodeTracer: Towards Traceable Agent States
| Authors | Han Li et al. |
| Year | 2026 |
| HF Upvotes | 39 |
| arXiv | 2604.11641 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks, making the agent's state transitions and error propagation hard to observe. In these runs, an early misstep can trap the agent in unproductive loops or even cascade into fundamental errors, forming hidden error chains that make it hard to tell when the agent goes off track and why. Existing agent tracing analyses either focus on simple interaction or rely on small-scale manual inspection, which limits their scalability and usefulness for real coding workflows. We present CodeTracer, a tracing architecture that parses heterogeneous run artifacts through evolving extractors, reconstructs the full state transition history as a hierarchical trace tree with persistent memory, and performs failure onset localization to pinpoint the failure origin and its downstream chain. To enable systematic evaluation, we construct CodeTraceBench from a large collection of executed trajectories generated by four widely used code agent frameworks on diverse code tasks (e.g., bug fixing, refactoring, and terminal interaction), with supervision at both the stage and step levels for failure localization. Experiments show that CodeTracer substantially outperforms direct prompting and lightweight baselines, and that replaying its diagnostic signals consistently recovers originally failed runs under matched budgets. Our code and data are publicly available.
Engineering Breakdown
Plain English
CodeTracer addresses a critical debugging gap in AI code agents: as these systems orchestrate parallel tool calls and multi-stage workflows, their internal state transitions become opaque, making it nearly impossible to trace why an agent fails or enters unproductive loops. The paper presents CodeTracer, a tracing architecture that parses heterogeneous run artifacts (logs, API calls, intermediate states) through evolving extractors to reconstruct the full execution graph and identify error chains. This enables engineers to see exactly when and why an agent goes off track, moving beyond manual inspection of small-scale runs to scalable observability of complex, real-world coding workflows.
Core Technical Contribution
The core innovation is a stateful artifact extraction system that reconstructs agent execution graphs from heterogeneous sources without requiring agents to emit standardized logs. Rather than asking agents to emit structured traces, CodeTracer reverse-engineers execution state by parsing tool calls, function returns, API responses, and memory snapshots through learned extractors that adapt to different agent frameworks and implementations. This approach solves the fundamental problem that real code agents don't emit clean, standardized traces—they produce messy, framework-specific artifacts that existing tools cannot parse at scale. The key technical novelty is the evolving extractor pattern, which learns to handle new agent architectures and error patterns through incremental updates rather than requiring manual instrumentation.
How It Works
CodeTracer operates in three stages: (1) Ingestion—it collects heterogeneous run artifacts from a code agent execution (tool call logs, function outputs, intermediate memory states, error messages) without requiring the agent framework to emit standardized telemetry. (2) Extraction and Reconstruction—evolving extractors (likely neural or pattern-based models) parse these artifacts and construct a state transition graph, building a timeline of what the agent's internal state was at each step. (3) Analysis—the reconstructed graph is analyzed to identify error propagation chains, detect where the agent diverged from optimal behavior, and pinpoint which tool call or state transition introduced the problem. The extractors themselves improve over time by observing new agent types and error patterns, allowing CodeTracer to handle diverse frameworks (ReAct, Tree-of-Thought, tool-use patterns) without manual rewriting.
Production Impact
For teams running code agents in production, CodeTracer transforms debugging from manual log inspection (hours per failure) to automated state reconstruction (seconds), making it feasible to diagnose failures across thousands of agent runs. A typical impact: instead of re-running a failed coding task with debug prints, engineers immediately see the execution graph showing that tool call N returned unexpected output, causing the agent to enter a loop attempting calls M through Q that should have been skipped. Integration cost is moderate—CodeTracer attaches as an external observer to existing agents without requiring framework changes, though setting up artifact collection for your specific tools adds engineering overhead. The main trade-off is storage: storing full execution graphs for every agent run can consume 10-100MB per run depending on task complexity, requiring log aggregation infrastructure. However, this cost is justified by the time saved in production debugging, especially for high-stakes systems where agent failures have business impact.
Limitations and When Not to Use This
CodeTracer assumes that sufficient signal exists in run artifacts to reconstruct agent state—if an agent operates mostly in-memory without emitting logs or API calls, reconstruction becomes incomplete. The paper's abstract doesn't specify whether the approach handles agents with stochastic or non-deterministic internal decisions well, or whether extractors degrade gracefully when encountering entirely new agent architectures. Extractors require training data from previous agent runs, creating a cold-start problem for newly deployed agents or rare failure modes that haven't been observed before. The scalability of the extractor itself (inference latency and memory for large execution graphs) is not discussed; for real-time debugging, extractors may need to run asynchronously, delaying problem identification.
Research Context
This work builds on a decade of research in program tracing, debuggers, and distributed systems observability (Dapper, Jaeger, OpenTelemetry), but applies these lessons to the new challenge of agent-based systems where the "program" is dynamically generated by an LLM. It complements concurrent work on agent interpretability and mechanistic interpretability by providing a coarser-grained execution-level view rather than probing model internals. CodeTracer opens a research direction toward self-improving observability: extractors that learn from failures could feed back into agent training, creating systems that become easier to debug over time. The work is timely given the explosion of code agents (GitHub Copilot X, Claude with tool use, OpenAI's code interpreter), where production debuggability remains a major unsolved problem.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::