Can Coding Agents Reproduce Findings in Computational Materials Science?
| Authors | Ziyang Huang et al. |
| Year | 2026 |
| Field | AI / ML |
| arXiv | 2605.00803 |
| Download | |
| Categories | cs.SE, cs.AI, cs.CL |
Abstract
Large language models are increasingly deployed as autonomous coding agents and have achieved remarkably strong performance on software engineering benchmarks. However, it is unclear whether such success transfers to computational scientific workflows, where tasks require not only strong coding ability, but also the ability to navigate complex, domain-specific procedures and to interpret results in the context of scientific claims. To address this question, we present AutoMat, a benchmark for evaluating LLM-based agents' ability to reproduce claims from computational materials science. AutoMat poses three interrelated challenges: recovering underspecified computational procedures, navigating specialized toolchains, and determining whether the resulting evidence supports a claim. By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow needed to support (or undermine) such claims. We then evaluate multiple representative coding agent settings across several foundation models. Our results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%. Error analysis further reveals that agents perform worst when workflows must be reconstructed from paper text alone and that they fail primarily due to incomplete procedures, methodological deviations, and execution fragility. Taken together, these findings position AutoMat as both a benchmark for computational scientific reproducibility and a tool for diagnosing the current limitations of agentic systems in AI-for-science settings.
Engineering Breakdown
Plain English
This paper introduces AutoMat, a benchmark designed to test whether large language models can work autonomously as coding agents to reproduce scientific findings in computational materials science. The authors identify a critical gap: while LLMs excel at general software engineering benchmarks, they struggle with domain-specific scientific workflows that require navigating complex toolchains, interpreting underspecified procedures, and validating whether computational results actually support the original scientific claims. AutoMat poses three concrete challenges to agents: recovering incomplete computational procedures from papers, working with specialized materials science tools, and determining claim-evidence alignment. The work benchmarks current LLM agents against these tasks to establish a baseline for future research.
Core Technical Contribution
The core contribution is the AutoMat benchmark itself—the first systematic evaluation framework for assessing LLM-based agents on scientific reproducibility rather than generic coding tasks. Unlike existing software engineering benchmarks (like HumanEval or SWE-Bench) that focus on isolated functions or self-contained repositories, AutoMat requires agents to operate end-to-end within real computational materials science workflows with missing specifications and domain-specific tools. The benchmark decomposes the reproducibility problem into three measurable subtasks: procedure recovery (inferring computational steps from incomplete paper descriptions), toolchain navigation (executing domain-specific software), and claim validation (determining whether outputs support the original hypothesis). This three-part structure captures the unique challenges of scientific work that generic code benchmarks miss entirely.
How It Works
AutoMat operates as follows: (1) The input is a scientific claim from a materials science paper along with partial or underspecified details about the computational procedure used. (2) An LLM-based agent must first recover the missing computational steps by reasoning about the scientific domain, inferring likely parameters, and reconstructing the workflow from sparse textual descriptions. (3) The agent then executes the procedure using specialized materials science toolchains (likely packages for density functional theory, molecular dynamics, or crystal structure prediction). (4) During execution, the agent must handle tool-specific APIs, error handling, and intermediate result interpretation. (5) Finally, the agent must analyze the numerical outputs and determine whether they support the original scientific claim with appropriate statistical rigor and physical intuition. (6) Evaluation measures success at each stage: correctness of recovered procedures, successful execution without errors, and accuracy of claim validation. The benchmark likely tests agents on multiple real papers with varying degrees of specification completeness to measure robustness.
Production Impact
For engineers deploying LLM agents in scientific computing contexts, AutoMat provides a concrete evaluation framework to assess whether models can actually reproduce research findings—a hard requirement for scientific software and computational chemistry/materials platforms. In production, this means: (1) Before integrating an LLM agent into a scientific workflow, you can benchmark it on AutoMat-like tasks to predict real-world reproducibility success; (2) The three-part challenge structure (procedure recovery, toolchain execution, claim validation) maps directly to failure modes in production—you now know where agents fail systematically; (3) Organizations building scientific automation platforms (think automated materials discovery pipelines, crystal structure prediction systems, or drug discovery workflows) can use this to quantify whether LLM assistance is reliable enough for their use case or whether human experts must still validate all outputs. The trade-off is clear: LLM agents are faster than manual reproduction but may recover incorrect procedures or misinterpret results, requiring human-in-the-loop validation that adds latency but prevents publishing false claims.
Limitations and When Not to Use This
The paper explicitly focuses on materials science, so findings may not generalize to other computational science domains (biology, climate, quantum simulation) that have different toolchain complexity and result interpretation standards. AutoMat assumes access to the original papers and some level of procedural transparency; it doesn't address cases where methods are deliberately proprietary or where reproducibility is impossible due to non-deterministic hardware/software interactions. The benchmark likely doesn't test agents on the most difficult real-world scenario: when a paper's description is so vague that multiple valid procedures could produce the claims, forcing the agent to make judgment calls about which interpretation is most likely—a task requiring deep domain expertise even humans struggle with. Additionally, the paper doesn't fully address how to evaluate partial successes (an agent that recovers 80% of the procedure correctly but simulates wrong boundary conditions) or how sensitive the final claim validation is to small numerical errors in intermediate steps.
Research Context
This work builds on two converging research directions: (1) the recent success of LLM-based coding agents (exemplified by works like SWE-Bench showing strong performance on repository-level tasks), and (2) the reproducibility crisis in computational science where even published papers often cannot be reproduced due to missing implementation details. AutoMat fills a specific gap identified in the broader "AI for Science" research community—previous work focused on using AI to generate novel scientific hypotheses or optimize existing known procedures, but AutoMat focuses on the preceding step: can AI agents reliably reproduce what humans already published? The benchmark also advances the evaluation methodology for scientific AI by moving beyond pure accuracy metrics to include interpretability and validity checks. This opens a new research direction: developing LLM agents that aren't just accurate at coding, but scientifically sound—agents that understand when numerical results contradict theoretical expectations or when a procedure can't possibly work given physical constraints.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::