InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
| Authors | Oliver Bentham & Vivek Srikumar |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.13201 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 45% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.
Engineering Breakdown
Plain English
This paper introduces InfiniteScienceGym, a procedurally generated benchmark designed to evaluate how well large language models can reason from empirical scientific data. Instead of relying on static datasets derived from published studies (which suffer from publication bias, label noise, and storage overhead), the authors built a simulator that deterministically generates unlimited numbers of self-contained scientific repositories with realistic directory structures, files, and tabular data. The system produces both answerable and unanswerable questions with exact ground truth, enabling rigorous evaluation of LLM capabilities in evidence-grounded reasoning, knowing when to abstain, and using tools to analyze data. This approach eliminates many artifacts of human-annotated benchmarks while providing infinite test diversity.
Core Technical Contribution
The core innovation is a procedurally generative benchmark system that produces infinite, verifiable scientific evaluation scenarios without manual annotation or publication bias. Rather than curating fixed datasets, the authors built a deterministic simulator seeded from simple parameters that generates complex, realistic scientific repositories complete with directory hierarchies, code files, and tabular datasets—then automatically produces paired QA instances with provably correct answers. This is fundamentally different from prior benchmarks (like SciEval or MATH) which are finite, static, and often polluted by bias from their source materials. The key insight is that procedural generation lets you scale evaluation infinitely while maintaining perfect ground truth, which is impossible with human annotation.
How It Works
The system operates in two stages: repository generation and question generation. First, given a seed value, the deterministic simulator generates a complete scientific repository with a realistic file tree, including CSV/tabular data files, Python scripts, documentation, and metadata. This generated repository is self-contained and reproducible—running the same seed always produces identical output. Second, a privileged QA generator that has access to the ground truth (the actual data and computation results) synthesizes both answerable questions (whose answers can be derived from the repository contents) and unanswerable questions (which require information absent from the repository). For each question, the system records the exact answer and the reasoning path, providing gold-standard labels. An LLM is then evaluated on its ability to answer correctly when the answer exists, refuse gracefully on unanswerable queries, and optionally execute analysis tools (like data queries) to derive answers.
Production Impact
For teams building LLM-based scientific assistants or code-understanding systems, this approach solves a critical evaluation problem: you can now measure reasoning on fresh, unseen data distributions without the labor of manual annotation or the biases baked into published papers. In production, you could adopt procedural benchmarking to continuously test model updates against infinite test sets with zero label noise, detecting regressions in tool use, abstention, and reasoning accuracy. The trade-off is implementation complexity—you must build the domain-specific simulator (not trivial for novel scientific domains), but the payoff is unlimited, unbiased evaluation data and the ability to scale evaluation to catch subtle model degradations. For data-hungry organizations, this eliminates the annotation bottleneck; for safety-conscious teams, the explicit ground truth enables precise measurement of hallucination and refusal behavior.
Limitations and When Not to Use This
The procedural generation quality is entirely dependent on the realism and diversity of the underlying simulator—if the generator produces overly simplified or stereotypical repositories, the benchmark won't transfer to real-world scientific data. The approach also assumes you can build a deterministic simulator for your specific domain (e.g., chemistry, biology, physics), which requires domain expertise and may not be practical for nascent or highly specialized fields. The paper doesn't address how well benchmarks trained on procedurally generated data generalize to real published datasets with genuine publication bias and human annotation quirks. Additionally, the reliance on a privileged QA generator (which has full access to ground truth) doesn't capture the challenge of evaluating reasoning on partially observable or ambiguous scientific data, which is common in practice.
Research Context
This work builds on a growing trend of procedurally generated benchmarks in AI (e.g., CLEVR for vision, Montezuma's Revenge for RL) but is the first large-scale application to scientific reasoning with LLMs. It responds directly to documented issues with existing scientific benchmarks—BioBench, SciEval, and others inherit biases from their source publications and plateau quickly as models improve. The paper positions itself as complementary to human-annotated benchmarks, offering unlimited, noise-free evaluation to complement real-world validation. This opens a research direction where procedural generation handles scale and cleanliness while separate real-world evaluations handle transfer and practical utility.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::