Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3
| Authors | Natapong Nitarach |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2603.27844 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Majority voting over multiple LLM attempts improves mathematical reasoning, but correlated errors limit the effective sample size. A natural fix is to assign different reasoning strategies to different voters. The approach, Diverse Prompt Mixer, is tested on the AIMO 3 competition: 3 models, 23+ experiments, 50 IMO-level problems, one H100 80 GB, 5-hour limit. Every prompt-level intervention fails. High-temperature sampling already decorrelates errors; weaker strategies reduce accuracy more than they reduce correlation. Across an 8-point capability gap at equal N=8 and every optimization tested, model capability dominates. The gap between the best majority-vote score (42/50) and pass@20 (~45.5) is selection loss, not prompt loss. A verifier-based selector could close it. Prompt engineering cannot.
Engineering Breakdown
Plain English
This paper investigates how to improve mathematical reasoning in large language models on AIMO 3 (a competition with 50 IMO-level problems) by combining multiple model attempts through voting and diverse prompting strategies. The authors tested majority voting across 3 models with 23+ experiments on a single H100 GPU in 5 hours, finding that every prompt-level optimization they tried failed to improve results. The key finding is that model capability—the raw strength of the underlying model—completely dominates any inference-time trick: an 8-point capability gap persisted even when they controlled for the number of attempts (N=8). The best majority-vote approach achieved 42/50, but simple pass@20 sampling achieved ~45.5, suggesting the real bottleneck is selecting which attempt to use, not generating better attempts through prompting.
Core Technical Contribution
The core contribution is an empirical finding that challenges the common assumption in LLM research that inference-time optimization (prompt engineering, diverse reasoning strategies, majority voting) can substantially improve mathematical reasoning. The authors introduce Diverse Prompt Mixer as a test case—assigning different reasoning prompts to different voters to decorrelate errors—but more importantly demonstrate through rigorous controlled experiments that this approach and similar prompt-level interventions fail. The key insight is that high-temperature sampling already provides sufficient error decorrelation, and any reduction in error correlation from strategic prompting comes at the cost of lower per-attempt accuracy, yielding no net gain. This reframes the problem: the bottleneck is not how well you can generate multiple diverse solutions, but rather how well you can select the correct one from candidates.
How It Works
The methodology starts with testing majority voting over multiple LLM attempts (a standard technique where you generate N solutions and take the most-agreed-upon answer). Typically, this works because individual attempts make different errors, so voting filters out noise. However, when errors are correlated—when models fail in similar ways—the effective sample size shrinks. The Diverse Prompt Mixer approach tries to break this correlation by giving different prompts to different models (e.g., 'solve step-by-step,' 'work backwards,' 'check your answer'), reasoning that diverse strategies would produce diverse errors. The authors implemented this across 3 models with 23+ experimental configurations on 50 problems within a 5-hour, single-GPU budget. They measured correlation of errors, individual model accuracy, majority-vote accuracy, and pass@k performance across different k values. The key finding emerged from comparing the gap between majority-vote accuracy (42/50) and pass@20 accuracy (~45.5)—this gap represents 'selection loss,' the benefit you'd get if you had a perfect oracle selector.
Production Impact
For engineers building mathematical reasoning systems in production, this paper delivers a sobering but actionable message: do not invest heavily in prompt engineering and diverse prompting strategies at inference time expecting major gains on hard reasoning tasks. Instead, invest in model pre-training quality and capability scaling, since a larger or better-trained base model will outperform any inference-time trick on a fixed compute budget. This suggests a shift in resource allocation: if you have a fixed inference-time budget (5 hours on one GPU), spending it on majority voting (generate many attempts, vote) is better than spending it on elaborate prompting or selection heuristics. However, the paper also points to a concrete actionable path forward: build a verifier model that can select which solution is correct without re-solving the problem. This could close the 3.5-point gap between majority voting (42/50) and pass@20 (45.5). In a production system, this means: scale your base model first, use simple majority voting with high-temperature sampling for diversity, and then invest in a separate verifier or selection component if you need further gains.
Limitations and When Not to Use This
The experiments are narrowly scoped: 50 IMO-level problems is a small evaluation set, and results may not generalize to broader reasoning tasks, multi-step planning, or domains outside competitive mathematics where error correlation and prompting effects may differ. The paper tests only 3 models on a single GPU with a 5-hour limit, so results may not hold for systems with more models, longer time horizons, or different hardware constraints where different optimization trade-offs emerge. The claim that 'prompt engineering fails' is strong but somewhat tautological: the paper doesn't test exhaustive prompt spaces, and future more-sophisticated prompting strategies might succeed where basic templates failed—the paper only disproves a specific set of prompt variants, not the entire space. Finally, the paper assumes that majority voting and pass@k are the right metrics, but doesn't explore other inference-time selection mechanisms (e.g., confidence-based selection, learned rankers, or hybrid approaches) that might leverage prompting more effectively than simple voting.
Research Context
This work builds on a long line of research using majority voting and ensemble methods to improve LLM reasoning, including chains-of-thought prompting and its variants. It engages directly with AIMO 3, an IMO-level benchmark for mathematical reasoning, which is one of the hardest reasoning tasks in LLM evaluation (more difficult than standard math benchmarks like GSM8K or MATH). The paper contributes to the emerging field of inference-time optimization and test-time compute scaling—the question of how to best use additional compute at inference time rather than training time. The finding that model capability dominates over prompt engineering complements concurrent research on scaling laws and suggests that for hard reasoning tasks, the return on investment for scaling base models exceeds the return for elaborate inference-time strategies. This opens a research direction: what are the properties of tasks where inference-time optimization does work (easier tasks, tasks with better prompt sensitivity) versus where it doesn't (hard, well-defined tasks like IMO problems)?
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::