MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
| Authors | Shaden Alshammari et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.18584 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts. MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that retrieval-augmented generation performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mathnet.mit.edu.
Engineering Breakdown
Plain English
This paper introduces MathNet, a large-scale dataset of 30,676 expert-authored Olympiad-level math problems spanning 47 countries, 17 languages, and 20 years of competitions. The authors created a comprehensive benchmark for evaluating both mathematical reasoning in generative models (like large language models) and mathematical retrieval in embedding-based systems (like semantic search). The dataset includes diverse problem domains and a carefully curated retrieval benchmark with mathematically equivalent and structurally similar problem pairs validated by human experts. This addresses a critical gap in existing benchmarks, which are too small, lack language diversity, and don't cover the full range of mathematical problem types needed to rigorously test AI reasoning capabilities.
Core Technical Contribution
The core innovation is constructing a high-quality, large-scale, multimodal, and genuinely multilingual mathematical problem dataset that enables dual evaluation paradigms: generative reasoning (end-to-end problem solving) and retrieval-based reasoning (finding similar problems). Unlike prior math benchmarks that focus narrowly on specific problem types or languages, MathNet systematically captures 30,676 problems across 47 countries and 17 languages, creating unprecedented diversity in problem structure, difficulty, and cultural context. The retrieval benchmark component—consisting of expert-curated pairs of mathematically equivalent and structurally similar problems—enables evaluation of embedding-based systems that currently lack adequate evaluation frameworks. This dual-benchmark approach recognizes that modern AI systems use both generative (LLMs) and retrieval (semantic embeddings) pathways for problem solving, requiring evaluation infrastructure for both.
How It Works
The dataset construction process begins with sourcing Olympiad-level math competition problems from multiple decades and countries, ensuring authentic expert-authored content rather than synthetically generated problems. Each problem is multimodal, containing both textual descriptions and mathematical notation/diagrams, and is paired with expert-authored solutions that explain the reasoning step-by-step. The retrieval benchmark is constructed through a two-stage human expert curation process: experts first identify mathematically equivalent problems (same solution approach, different presentation), then identify structurally similar problems (different solution approach but similar problem structure). For the generative reasoning benchmark, problems are presented to language models and multimodal models, with evaluation based on correct solution derivation. For the retrieval benchmark, embedding systems must rank candidate problems by semantic similarity to a query problem, with relevance determined by the expert-curated equivalence pairs.
Production Impact
For teams building AI-assisted math tutoring or homework systems, MathNet provides a rigorous evaluation framework to measure reasoning quality in both code-based solvers (generative) and retrieval-augmented systems (embedding-based). This enables production teams to benchmark new models against a genuinely challenging, multilingual dataset that reflects real-world problem diversity rather than synthetic benchmarks. The multimodal and multilingual nature is particularly valuable for companies expanding math AI globally—you can now test language-specific and culturally-specific problem understanding. However, production adoption requires significant infrastructure: the dataset is large (30,676 problems), and evaluation is computationally expensive (Olympiad-level problems require multiple reasoning steps). Teams should expect 3-5x slower evaluation compared to simpler math benchmarks, and the expert-curated retrieval pairs mean this isn't suitable for on-the-fly synthetic expansion.
Limitations and When Not to Use This
The paper focuses on Olympiad-level problems, which represent an artificial ceiling on difficulty and may not reflect the distribution of math problems encountered in real educational or professional settings. Olympiad problems emphasize elegant proofs and insight-based solutions rather than procedural computation, potentially missing evaluation of other critical mathematical reasoning modes. The language coverage (17 languages from 47 countries) likely has significant imbalance—some languages probably have dozens of problems while others have hundreds, limiting the ability to fairly evaluate cross-lingual generalization. The paper doesn't address solution uniqueness or multiple valid solution paths, which are common in real mathematics, potentially creating evaluation brittleness if a model uses a valid but unrecognized solution approach.
Research Context
This work builds on prior math benchmarking efforts like GSM8K and MATH, which created synthetic or limited-scale datasets with weak multilingual coverage. MathNet improves on these by using authentic competition problems (avoiding synthetic artifacts), providing significantly larger scale with true language diversity, and introducing the retrieval evaluation dimension which existing benchmarks lack. The dataset advances the broader research direction of multimodal reasoning, where models must process both text and mathematical notation together—a requirement largely absent from prior benchmarks. This opens research opportunities in few-shot learning across languages (can models trained on English Olympiad problems transfer to Korean ones?) and in understanding whether semantic embeddings can capture mathematical equivalence, a fundamental question for retrieval-augmented math systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::