Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
| Authors | Wang Bill Zhu et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.17338 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
Engineering Breakdown
Plain English
This paper addresses a critical gap in how we evaluate code debugging in large language models. The authors observed that frontier LLMs often don't actually debug code precisely—they regenerate entire functions or make unnecessary edits even when the bug is localized and fixable with minimal changes. To measure this, they built the Precise Debugging Benchmark (PDB) framework, which converts any coding dataset into a rigorous debugging evaluation suite by synthetically injecting verified bugs and measuring both edit-level precision (how many unnecessary edits were made) and bug-level recall (how many bugs were actually fixed). The framework supports single-line bugs (PDB-Single-Hard) and multi-line bugs (PDB-Multi), providing the first standardized way to distinguish genuine debugging from lazy regeneration.
Core Technical Contribution
The core novelty is the precision-aware evaluation framework that reframes debugging as a constrained editing problem rather than code generation. Instead of accepting any correct output, PDB measures two new metrics: edit-level precision quantifies unnecessary changes (penalizing over-editing), and bug-level recall counts bugs resolved (ensuring completeness). The synthetic bug generation pipeline composes atomic, verified bugs into multi-bug programs, ensuring realistic debugging scenarios that can be systematically created from any existing code dataset. This is fundamentally different from prior benchmarks like HumanEval or MBPP, which only measure whether the final output is correct, not whether the model took an efficient, targeted path to fix the problem.
How It Works
The system operates in three stages. First, it takes a reference coding dataset and instruments each program with synthetic bugs by composing verified atomic bugs (single-line modifications known to break the code in specific ways). Second, the buggy program becomes the input to the LLM's debugging task, along with the original correct version and bug reports. Third, the LLM generates a fixed version, which is compared against the reference using two-level evaluation: at the edit level, the system counts how many lines the LLM changed versus the minimum necessary changes (precision = necessary edits / total edits), and at the bug level, it verifies which bugs from the original synthetic composition were actually resolved (recall = bugs fixed / bugs injected). The framework tracks both metrics separately because an LLM could achieve high bug-level recall by rewriting the entire function (low precision) or miss bugs while making targeted edits (lower recall).
Production Impact
This framework directly improves code review and automated bug-fixing pipelines by letting teams measure whether their LLM solutions are safe for production. In real deployment, over-editing code introduces unnecessary risk—it can break surrounding logic, fail code review processes, or trigger extensive regression testing. By adopting PDB evaluation, engineering teams can quantify whether their debugging LLM is surgically fixing bugs or lazily regenerating code, and make informed decisions about whether to deploy the system or fine-tune it further. The precision metric is particularly valuable for enterprise use cases where code changes must be minimal and reviewable; a high-precision, high-recall system could reduce manual code review burden and regression test scope. However, there are trade-offs: the synthetic bug injection process adds dataset preparation complexity, and the evaluation requires reference implementations and bug composition logic that may not generalize seamlessly to proprietary codebases without customization.
Limitations and When Not to Use This
The paper relies on synthetic bug injection, which may not capture the full distribution of real-world bugs encountered by developers (fat-finger typos, logic errors, and architectural flaws likely have different frequency and interaction patterns than algorithmically composed synthetic bugs). The evaluation assumes access to ground-truth reference implementations and the ability to verify which bugs were actually introduced, making this harder to apply to undocumented or legacy code. Edit-level precision assumes that line-by-line diffs are the right granularity for measuring unnecessary changes; in practice, LLMs might make semantically redundant changes within a line or refactor surrounding code in ways that are legitimate but penalized by the metric. The paper also doesn't address how to weight precision vs. recall trade-offs in production—a team must decide whether high precision with some missed bugs is better than perfect bug fixing with over-editing, and PDB doesn't provide guidance on setting that threshold.
Research Context
This work builds on a decade of code generation benchmarking (starting with HumanEval and MBPP) but identifies a blind spot: correctness alone doesn't capture whether LLMs are truly debugging or just lucky at regeneration. It aligns with recent industry focus on agentic AI and multi-step reasoning tasks, where intermediary steps and edit efficiency matter as much as final outputs. The precision metric is inspired by information retrieval and information extraction evaluation (standard precision/recall from NLP), adapted here to the code editing domain. The paper opens a research direction around controllable code generation—future work could explore whether LLMs can be fine-tuned or prompted to prioritize minimal edits, and whether pre-training on debugging-specific data (version control diffs, code review comments) improves precision.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::