Solving Physics Olympiad via Reinforcement Learning on Physics Simulators
| Authors | Mihir Prabhudesai et al. |
| Year | 2026 |
| HF Upvotes | 16 |
| arXiv | 2604.11805 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question-answer (QA) pairs, a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.
Engineering Breakdown
Plain English
This paper addresses a critical bottleneck in training reasoning-capable language models: the scarcity of large-scale question-answer datasets in physics compared to mathematics. Rather than relying on internet QA pairs, the authors propose using physics simulators as a synthetic data source, generating random scenes, creating question-answer pairs from simulated interactions, and training LLMs via reinforcement learning on this synthetic data. The key finding is that models trained this way exhibit strong zero-shot transfer capabilities to real physics problems, demonstrating that simulator-based supervision can be a viable alternative to scarce domain-specific QA datasets.
Core Technical Contribution
The core novelty is using physics engines as an automated, scalable source of supervision for training physical reasoning in LLMs, bypassing the need for human-curated QA datasets. The authors introduce a pipeline that generates synthetic physics scenarios, converts them into natural language question-answer pairs, and applies reinforcement learning to optimize model responses against ground-truth simulator outcomes. This is fundamentally different from prior work that either relies on limited internet QA data or trains models only on text; here, the simulator itself becomes the reward signal and teacher. The approach enables zero-shot generalization to unseen physics problems, suggesting that synthetic supervisory signals from simulators can encode physical principles effectively.
How It Works
The system operates in four main stages: (1) Random scene generation—the authors procedurally create diverse physics scenarios in simulation engines (e.g., object interactions, collisions, forces). (2) Synthetic QA pair creation—for each scene, the system generates natural language questions (e.g., 'What happens if we apply force X?') and ground-truth answers by running the simulation forward. (3) LLM training via RL—the model is fine-tuned using reinforcement learning where the reward is based on how well the model's predicted answer matches the simulator's outcome, using techniques similar to RLHF (Reinforcement Learning from Human Feedback) but with simulator feedback instead. (4) Evaluation—the trained model is tested on Physics Olympiad problems and real-world physics reasoning tasks. The key insight is that simulator trajectories provide dense, accurate labels automatically, avoiding manual annotation while covering a vast space of physical scenarios.
Production Impact
This approach could dramatically reduce the engineering effort required to build domain-specific reasoning systems in sciences where QA data is scarce—biology, chemistry, materials science, and engineering all could benefit. Instead of collecting thousands of expert-annotated QA pairs, teams could spin up a simulator (or use existing ones like Bullet, MuJoCo, or CoppeliaSim) and automatically generate unlimited training data. The trade-off is that your model's performance is capped by simulator fidelity; physics engines make approximations and don't capture all real-world phenomena, so zero-shot transfer on novel real-world problems remains challenging. Compute cost would shift from data curation to simulation and RL training—potentially lower for data generation but higher for RL iterations. Integration would require choosing an appropriate simulator for your domain and implementing a synthetic-to-natural-language conversion pipeline, which is domain-specific engineering work.
Limitations and When Not to Use This
The paper's evaluation is limited to physics problems; it's unclear whether this approach scales to other sciences with less mature simulators (e.g., biology, chemistry). Simulator fidelity is a hard ceiling—if your simulator doesn't model an effect, the model won't learn it, and real-world physics always has edge cases simulators miss. The zero-shot claims likely depend heavily on how well the training distribution of synthetic scenarios matches the test distribution; distributional shift between simulator and real problems could be severe. The paper doesn't adequately discuss what happens when simulators themselves have bugs or unrealistic approximations, and whether the RL training can robustly handle conflicting signals when the simulator is wrong. Additionally, the abstract is cut off and doesn't report actual benchmark numbers, making it hard to assess true performance gains versus baselines.
Research Context
This work builds on the recent success of reasoning models like DeepSeek-R1 and extends the paradigm of learning from synthetic data (already proven in vision via large-scale 3D simulators like CARLA and Omniverse). It complements RLHF and similar RL-from-feedback approaches by showing that non-human supervisors (physics engines) can provide high-quality training signals at scale. The paper opens a research direction toward simulator-based curriculum learning and synthetic data generation as a general alternative to internet-mined QA datasets, especially valuable for niche domains. It also relates to broader work on neural physics engines and differentiable simulation, though this paper uses simulators differently—as labelers rather than as part of the model architecture.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::