LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics
| Authors | Antoine Peyronnet et al. |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2602.24173 |
| Download | |
| Categories | cs.AI |
Abstract
We present a new approach for benchmarking Large Language Model (LLM) capabilities on research-level mathematics. Existing benchmarks largely rely on static, hand-curated sets of contest or textbook-style problems as proxies for mathematical research. Instead, we establish an updatable benchmark evaluating models directly on the latest research results in mathematics. This consists of an automatic pipeline that extracts lemmas from arXiv and rewrites them into self-contained statements by making all assumptions and required definitions explicit. It results in a benchmark that can be updated regularly with new problems taken directly from human mathematical research, while previous instances can be used for training without compromising future evaluations. We benchmark current state-of-the-art LLMs, which obtain around 10-15 accuracy in theorem proving (pass@1) depending on the model, showing that there is currently a large margin of progression for LLMs to reach human-level proving capabilities in a research context.
Engineering Breakdown
Plain English
LemmaBench introduces a live, continuously-updated benchmark for evaluating large language models on research-level mathematics, rather than relying on static contest problems. The core innovation is an automatic pipeline that extracts mathematical lemmas from arXiv papers and rewrites them into self-contained, explicit statements with all assumptions and definitions included. This allows the benchmark to grow with new research while keeping previous versions available for training without data leakage into future test sets. The work directly addresses a critical gap: existing math benchmarks use curated textbook or competition problems that don't represent the actual frontier of mathematical research where LLMs will be deployed.
Core Technical Contribution
The primary technical contribution is a fully automated extraction and reformulation pipeline that transforms informal mathematical research (arXiv lemmas) into standardized, self-contained benchmark problems suitable for systematic LLM evaluation. Unlike static benchmarks like MATH or competition datasets, this approach creates a living evaluation framework where new problems are continuously sourced from active research, ensuring the benchmark stays at the cutting edge of mathematical difficulty and relevance. The pipeline handles the non-trivial problem of making implicit assumptions explicit and providing necessary definitions—transformations that require understanding mathematical context and structure. This enables a virtuous cycle where older benchmark instances can safely be used for training without compromising the statistical validity of future evaluation sets, solving the cold-start and contamination problems that plague existing benchmarks.
How It Works
The system operates as a multi-stage pipeline: first, it automatically identifies and extracts lemma statements from arXiv mathematics papers using heuristics or learned patterns to locate formal mathematical claims. Second, the extracted lemmas are processed through a rewriting stage that makes all assumptions, preconditions, and required definitions explicit in a single self-contained statement—this is crucial because research papers assume reader familiarity with domain knowledge that an LLM may not have at inference time. Third, the reformulated lemmas are validated for clarity and correctness (likely with human review or consistency checks). Finally, the benchmark is versioned: new lemmas become part of the active test set while older instances are explicitly marked as suitable for training, creating temporal separation that prevents data leakage. The pipeline likely uses language model assistance to aid the extraction and reformulation steps, but the key insight is automating what has historically been manual human curation.
Production Impact
For teams building production math-capable AI systems, this benchmark provides a far more realistic evaluation surface than existing static datasets—it tells you whether your model can handle mathematics at the frontier of human research, not just textbook or competition problems. Adopting this for internal evaluation would mean integrating a continuous data pipeline from arXiv, which adds operational overhead but provides early warning signals as mathematical research progresses and becomes harder. The versioning mechanism is particularly valuable: you can safely train on yesterday's benchmark instances without worrying that your evaluation set is contaminated, enabling faster iteration and larger training runs. The trade-off is that this requires infrastructure for automated extraction, reformulation, and validation—you need to either run the full pipeline yourself or rely on updated versions from the authors, introducing a dependency on external data freshness. For companies evaluating LLM acquisition decisions, having a live benchmark aligned with actual research difficulty is significantly more credible than contest-based scores.
Limitations and When Not to Use This
The paper's approach assumes that mathematical research on arXiv is a good proxy for the actual distribution of problems you care about—but research mathematics emphasizes proof-based reasoning and specialized domains, which may not cover applied mathematics, numerical methods, or interdisciplinary uses. The automated extraction and reformulation pipeline introduces its own failure modes: you risk extracting malformed or misleading lemmas if the heuristics fail, or producing self-contained statements that are grammatically correct but mathematically nonsensical or harder than the original claim. Human validation at scale becomes a bottleneck; the paper doesn't clearly explain how validation is performed or what error rates exist in the reformulation process. Additionally, there's a question of benchmark stability: if arXiv becomes more or less rigorous over time, or if certain mathematical subfields suddenly surge in popularity, the benchmark's difficulty distribution could shift unpredictably, making year-over-year comparisons less meaningful.
Research Context
This work builds on a decades-long tradition of mathematical benchmarking (MATH dataset, competition math problems, textbook exercises) but fundamentally rejects the static curation model that dominates the field. It's motivated by observations from model scaling research showing that larger LLMs saturate existing benchmarks quickly, creating demand for harder and more realistic evaluation sets that represent open research frontiers. The approach also draws from recent work on automatically constructing evaluation sets and avoiding train-test contamination in the large-model era—concerns that became acute after several high-profile incidents where benchmark leakage inflated reported performance. The paper opens up a research direction around living benchmarks and real-time evaluation alignment, with potential applications beyond mathematics (theoretical physics, formal verification, symbolic reasoning) wherever human research is continuously published in structured formats.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::