AI scientists produce results without reasoning scientifically
| Authors | Martiño Ríos-García et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.18805 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously, yet whether their reasoning adheres to the epistemic norms that make scientific inquiry self-correcting is poorly understood. Here, we evaluate LLM-based scientific agents across eight domains, spanning workflow execution to hypothesis-driven inquiry, through more than 25,000 agent runs and two complementary lenses: (i) a systematic performance analysis that decomposes the contributions of the base model and the agent scaffold, and (ii) a behavioral analysis of the epistemological structure of agent reasoning. We observe that the base model is the primary determinant of both performance and behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Across all configurations, evidence is ignored in 68% of traces, refutation-driven belief revision occurs in 26%, and convergent multi-test evidence is rare. The same reasoning pattern appears whether the agent executes a computational workflow or conducts hypothesis-driven inquiry. They persist even when agents receive near-complete successful reasoning trajectories as context, and the resulting unreliability compounds across repeated trials in epistemically demanding domains. Thus, current LLM-based agents execute scientific workflows but do not exhibit the epistemic patterns that characterize scientific reasoning. Outcome-based evaluation cannot detect these failures, and scaffold engineering alone cannot repair them. Until reasoning itself becomes a training target, the scientific knowledge produced by such agents cannot be justified by the process that generated it.
Engineering Breakdown
Plain English
This paper evaluates whether large language model-based scientific agents actually reason scientifically when conducting autonomous research. The researchers ran over 25,000 agent executions across eight scientific domains, measuring both task performance and the epistemic quality of the reasoning process. They found that the underlying LLM itself accounts for 41.4% of performance variance while the agent scaffolding (prompts, tools, planning structures) contributes only 1.5%, suggesting that improvements to base models matter far more than engineering better agent workflows. The core finding is troubling: these systems often produce correct answers without following proper scientific reasoning patterns—they lack self-correction mechanisms, hypothesis validation, and transparent uncertainty quantification that characterize human scientific inquiry.
Core Technical Contribution
The paper's primary contribution is a dual-lens evaluation framework that separates base model performance from agent scaffold contributions in scientific reasoning tasks. Rather than just measuring task accuracy, the authors introduced a behavioral analysis methodology that examines the epistemological structure of agent reasoning—essentially scoring whether the agent's reasoning process exhibits properties like falsifiability testing, alternative hypothesis consideration, and uncertainty acknowledgment. They demonstrate this novel evaluation approach across eight scientific domains with unprecedented scale (25,000+ runs), providing empirical evidence that scaffolding improvements (which dominate recent agent research) have minimal impact compared to base model quality. This reframes the agent development narrative: engineering better prompts and tools provides marginal returns when the foundation model lacks scientific reasoning patterns.
How It Works
The evaluation framework operates in two parallel channels: performance analysis and behavioral epistemology analysis. In the performance channel, agents receive scientific tasks (hypothesis testing, workflow execution, etc.) and the authors decompose variance attribution using statistical techniques that isolate the base LLM's contribution from the agent scaffold's contribution (planning modules, tool integration, retrieval systems). The behavioral channel instrumentally audits agent reasoning transcripts against epistemic norms—does the agent test alternative hypotheses, acknowledge uncertainty bounds, iterate after negative results, or just pattern-match to plausible-sounding answers? Input workflows span eight domains; outputs are both success/failure metrics and qualitative reasoning traces. The key mechanism is that they hold domain tasks constant while varying which LLM and which scaffolding sits behind the agent, allowing them to measure independent effects of each component.
Production Impact
This research directly challenges the current engineering emphasis on agent scaffolding (RAG systems, tool-use frameworks, prompt engineering). Production teams investing heavily in better prompts, retrieval systems, and planning modules should expect marginal gains—the 1.5% variance attribution to scaffolding means that doubling scaffolding sophistication yields minimal performance lift. Instead, this suggests teams should prioritize upgrading to stronger base models, which account for 41.4% of explained variance. For scientific AI applications specifically (drug discovery, materials science, data analysis), this implies that accuracy alone is insufficient; you must also audit whether the agent's reasoning matches scientific epistemology—can it articulate why it rejected alternatives, what assumptions it made, what confidence bounds apply? This adds evaluation cost but prevents systems that look correct while reasoning incorrectly, which is especially risky in domains where reproducibility and transparency are regulatory or institutional requirements.
Limitations and When Not to Use This
The paper does not provide mechanisms to improve agents beyond upgrading base models—it diagnoses the problem but offers limited prescriptive solutions for teams constrained to fixed models. The behavioral epistemology analysis, while novel, lacks ground truth definitions of what constitutes 'proper' scientific reasoning across diverse domains; the scoring rubric may not transfer to specialized sciences (quantum physics, protein folding) where reasoning norms differ sharply. The evaluation is static—it measures agents at a single point in time rather than tracing how reasoning patterns evolve with in-context learning or fine-tuning, so it's unclear whether weak epistemic reasoning is fundamental or addressable through training. Finally, the 25,000 runs, while substantial, are distributed across eight domains, giving limited statistical power per domain; domain-specific conclusions should be treated cautiously.
Research Context
This work sits at the intersection of recent agent research (autonomous tool-using LLM systems that have proliferated since 2023) and philosophy of science applied to AI. It responds directly to the boom in scaffolding-focused agent papers—ReAct, Tree-of-Thought, self-refinement prompting—by empirically quantifying their actual contribution. The paper builds on prior work in AI interpretability and reasoning evaluation (mechanistic interpretability, chain-of-thought analysis) but applies it to the specific problem of scientific validity. It reinforces trends in foundation model research emphasizing base model quality (scaling laws, instruction tuning) over downstream task engineering, aligning with observations from recent benchmark work (e.g., MMLU scaling, SuperGLUE saturation). The research opens a new evaluation frontier: separating task performance from reasoning quality, which will likely influence how the community measures and builds scientific AI systems going forward.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::