SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?
| Authors | Udari Madhushani Sehwag et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.10718 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Accelerating scientific discovery requires the identification of which experiments would yield the best outcomes before committing resources to costly physical validation. While existing benchmarks evaluate LLMs on scientific knowledge and reasoning, their ability to predict experimental outcomes - a task where AI could significantly exceed human capabilities - remains largely underexplored. We introduce SciPredict, a benchmark comprising 405 tasks derived from recent empirical studies in 33 specialized sub-fields of physics, biology, and chemistry. SciPredict addresses two critical questions: (a) can LLMs predict the outcome of scientific experiments with sufficient accuracy? and (b) can such predictions be reliably used in the scientific research process? Evaluations reveal fundamental limitations on both fronts. Model accuracies are 14-26% and human expert performance is approx20%. Although some frontier models exceed human performance model accuracy is still far below what would enable reliable experimental guidance. Even within the limited performance, models fail to distinguish reliable predictions from unreliable ones, achieving only approx20% accuracy regardless of their confidence or whether they judge outcomes as predictable without physical experimentation. Human experts, in contrast, demonstrate strong calibration: their accuracy increases from approx5% to approx80% as they deem outcomes more predictable without conducting the experiment. SciPredict establishes a rigorous framework demonstrating that superhuman performance in experimental science requires not just better predictions, but better awareness of prediction reliability. For reproducibility all our data and code are provided at https://github.com/scaleapi/scipredict
Engineering Breakdown
Plain English
SciPredict is a new benchmark containing 405 scientific experiment prediction tasks spanning 33 specialized sub-fields across physics, biology, and chemistry. The authors evaluate whether large language models can predict experimental outcomes accurately enough to be useful in real scientific research, addressing a gap where current LLM benchmarks test knowledge and reasoning but not predictive capability on actual empirical studies. The paper investigates two critical questions: whether LLMs can achieve sufficient accuracy for experiment outcome prediction, and whether those predictions can be reliably integrated into the scientific research workflow. This work treats AI prediction of experimental results as a high-value application area where machines could potentially outperform human intuition, but the actual performance results and reliability assessment appear to be cut off in the provided abstract.
Core Technical Contribution
The core contribution is the construction and release of SciPredict, a curated benchmark specifically designed to evaluate LLMs on experiment outcome prediction rather than static knowledge retrieval or general reasoning. Unlike existing scientific benchmarks that focus on multiple-choice questions, mathematical problem-solving, or document comprehension, SciPredict grounds evaluation in real empirical studies with documented outcomes, forcing models to make concrete predictive claims about what happens when specific experimental conditions are applied. The benchmark's design across 33 specialized sub-fields ensures breadth of scientific domains while maintaining rigor through grounding in actual published research. By framing the evaluation around two distinct capability questions—accuracy and practical reliability—the authors establish a framework for assessing whether LLM predictions are suitable for integration into real scientific workflows.
How It Works
The benchmark construction process takes recent empirical studies from physics, biology, and chemistry and derives 405 distinct prediction tasks by extracting experimental setups and their documented outcomes. Each task likely presents a model with a description of experimental parameters, materials, methods, or conditions, then evaluates whether the model can predict the resulting outcome that was observed in the published study. The evaluation framework operates on two levels: first, it measures prediction accuracy using metrics appropriate to each task type (likely including classification accuracy, regression error, or ranking metrics depending on whether outcomes are discrete or continuous). Second, the framework assesses reliability by examining whether model confidence correlates with correctness and whether predictions could actually guide resource allocation decisions in practice. This two-tier evaluation distinguishes between raw predictive ability and trustworthiness for downstream use.
Production Impact
For engineers building AI-assisted scientific discovery systems, SciPredict provides a validation mechanism for whether foundation models can meaningfully contribute to experiment planning and design. In a production pipeline, accurate experiment outcome prediction could significantly reduce the cost and time of scientific research by allowing researchers to prioritize high-probability-of-success experiments before committing expensive computational resources, lab time, or materials. Integration would involve building a prediction service that takes experimental designs as structured input and returns outcome predictions with calibrated confidence scores, then connecting this to experiment management systems that use predictions for automated prioritization. However, the limitation becomes critical here: if LLM predictions lack reliability (confidence miscalibration or systematic biases in specific domains), deploying them could waste resources on confident but wrong predictions, so production systems would require extensive validation on domain-specific test sets and human-in-the-loop review for high-stakes decisions. The compute cost of running LLM inference on potentially thousands of candidate experiments per research project is substantial, making batch prediction and caching strategies important for cost management.
Limitations and When Not to Use This
The abstract does not provide actual performance metrics, so it remains unknown whether LLMs achieve practically useful accuracy on this benchmark—they may perform only marginally better than baselines or random guessing, particularly in highly specialized sub-fields. The benchmark is derived from published studies, which may introduce selection bias toward experiments that succeeded or produced interesting results, making the outcome distribution non-representative of realistic experiment planning where many attempts fail. LLMs may also suffer from domain-specific knowledge gaps in specialized sub-fields of physics, biology, and chemistry, leading to poor performance when predicting outcomes in domains with limited training data representation. The paper does not appear to address whether models can explain their predictions in ways scientists would find trustworthy, nor whether predictions remain reliable when experimental conditions are slightly perturbed—both critical for real-world adoption where researchers need interpretability and robustness to measurement uncertainty.
Research Context
This work builds on the growing body of scientific benchmarking for LLMs (such as SciBench, STEMQA, and others that evaluate knowledge), but shifts focus from retrieval and reasoning to forward prediction of empirical outcomes. The benchmark extends the paradigm established by recent work on task-specific LLM evaluation by grounding assessment in real published science rather than synthetic questions or standardized tests. SciPredict opens a research direction around outcome prediction as a distinct capability separate from knowledge, potentially motivating follow-up work on domain-specific fine-tuning, retrieval-augmented generation over experimental databases, and ensemble approaches combining LLMs with physics-based or statistical models. This work also intersects with broader discussions in AI for science about whether foundation models can meaningfully accelerate discovery, or whether domain-specific methods remain necessary for reliable prediction.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::