Do AI Coding Agents Log Like Humans? An Empirical Study
| Authors | Youssef Esseddiq Ouatiti et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.09409 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Software logging is essential for maintaining and debugging complex systems, yet it remains unclear how AI coding agents handle this non-functional requirement. While prior work characterizes human logging practices, the behaviors of AI coding agents and the efficacy of natural language instructions in governing them are unexplored. To address this gap, we conduct an empirical study of 4,550 agentic pull requests across 81 open-source repositories. We compare agent logging patterns against human baselines and analyze the impact of explicit logging instructions. We find that agents change logging less often than humans in 58.4% of repositories, though they exhibit higher log density when they do. Furthermore, explicit logging instructions are rare (4.7%) and ineffective, as agents fail to comply with constructive requests 67% of the time. Finally, we observe that humans perform 72.5% of post-generation log repairs, acting as "silent janitors" who fix logging and observability issues without explicit review feedback. These findings indicate a dual failure in natural language instruction (i.e., scarcity of logging instructions and low agent compliance), suggesting that deterministic guardrails might be necessary to ensure consistent logging practices.
Engineering Breakdown
Plain English
This paper investigates how AI coding agents handle software logging compared to humans, analyzing 4,550 pull requests across 81 open-source repositories. The researchers found that agents change logging less frequently than humans in 58.4% of repositories, but when they do log, they tend to add more log statements per change. A key finding is that explicit natural language instructions about logging are rare (appearing in only 4.7% of cases) and largely ineffective at steering agent behavior, suggesting that current AI agents struggle to follow domain-specific non-functional requirements even when explicitly told.
Core Technical Contribution
The core contribution is the first empirical characterization of how AI coding agents handle logging, a critical but understudied non-functional requirement in software systems. Unlike prior work that documents human logging practices in isolation, this paper directly compares agent behavior against human baselines at scale, revealing systematic differences in logging frequency and density. The authors also introduce and evaluate the effectiveness of natural language instructions as a mechanism for governing agent logging behavior, discovering that instruction-based steering has minimal impact on actual agent outputs.
How It Works
The study methodology involves mining 4,550 agentic pull requests from 81 open-source GitHub repositories and extracting logging-related changes (additions, deletions, modifications of log statements). For each pull request, the researchers compare logging patterns between the agent-generated code and human-generated baseline commits by measuring log density (number of log statements per code change) and change frequency. They then identify cases where explicit logging instructions appear in commit messages or comments and measure whether these instructions correlate with improved logging behavior. The analysis uses quantitative metrics (e.g., percentage of repos where agents log less than humans) and qualitative examination of instruction effectiveness to determine whether natural language directives can steer agent logging decisions.
Production Impact
For teams deploying AI coding agents in production systems, this research reveals a critical gap: agents cannot reliably produce adequate logging infrastructure even with explicit instructions, which directly impacts system observability and debuggability. This means engineers adopting AI agents should expect to manually audit and augment logging in agent-generated code, adding post-hoc review cycles to the CI/CD pipeline rather than trusting agents to handle this requirement end-to-end. The finding that agents exhibit higher log density when they do log suggests potential over-logging in some cases, which could increase storage costs and reduce signal-to-noise in observability stacks. Production teams should build automated checks to validate logging completeness against organization-specific logging standards and potentially use techniques like few-shot examples or training-time adjustments rather than relying on natural language prompts to steer logging behavior.
Limitations and When Not to Use This
The paper studies logging behavior in isolation without analyzing whether insufficient agent logging actually impacts system reliability or debuggability in practice—correlation with downstream observability failures is not established. The study is limited to open-source repositories, which may have different logging conventions and agent instruction patterns than closed-source enterprise codebases where logging requirements are often more strict and well-defined. The research does not explore whether alternative steering mechanisms (e.g., code templates, in-context examples, fine-tuned models) are more effective than natural language instructions, leaving open the question of whether the problem is fundamentally unsolvable for current agents or simply a matter of better prompting. Additionally, the paper does not distinguish between different types of AI agents or models, so findings may not generalize across different agent architectures or foundation models.
Research Context
This work builds on prior research characterizing human logging practices and software maintenance patterns while extending it into the emerging area of AI agent behavior. It complements existing literature on AI code generation quality (e.g., studies of correctness, test coverage, security) by focusing on an understudied non-functional aspect that is critical for production systems. The paper opens a research direction around how to make AI agents respect non-functional requirements and domain-specific software engineering practices, which is timely given the rapid adoption of AI coding assistants like GitHub Copilot and specialized agents in software development workflows.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::