Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints
| Authors | Xinge Liu et al. |
| Year | 2026 |
| HF Upvotes | 4 |
| arXiv | 2604.15664 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of these agents in research work. We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks using inference on radial-velocity (RV) time series data. Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases, covering diverse scenarios ranging from high-SNR single-planet systems to complex multi-planetary configurations requiring involved low-SNR analysis. Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints: although agents often achieve a good statistical fit, they frequently fail to recover correct physical system parameters, a limitation that persists even when agents are equipped with vanilla skills. Furthermore, increasing test-time compute yields only marginal gains, with excessive token usage often reflecting recursive failure loops rather than meaningful exploration. Stargazer presents an opportunity to train, evaluate, scaffold, and scale strategies on a model-fitting problem of practical research relevance today. Our methodology to design a simulation-driven environment for AI agents presumably generalizes to many other model-fitting problems across scientific domains. Source code and the project website are available at https://github.com/Gudmorning2025/Stargazer and https://gudmorning2025.github.io/Stargazer, respectively.
Engineering Breakdown
Plain English
Stargazer is a benchmark environment designed to evaluate how well autonomous AI agents can perform physics-based research tasks, specifically fitting models to radial-velocity time series data to detect exoplanets. The authors created 120 tasks across three difficulty levels, including 20 real archival cases, ranging from simple single-planet systems to complex multi-planet configurations with low signal-to-noise ratios. Testing eight frontier AI agents revealed a critical gap: while agents often optimize numerical loss functions well, they frequently violate fundamental physical constraints that real scientists would never ignore. This work establishes a scalable framework for evaluating whether AI agents can do genuine scientific work, not just mathematical optimization.
Core Technical Contribution
The core novelty is a scientifically grounded, dynamic benchmark environment that moves beyond standard ML benchmarks by embedding domain-specific physical constraints and real archival data into agent evaluation tasks. Rather than static classification or regression problems, Stargazer requires agents to engage in iterative refinement loops—collecting observations, fitting models, analyzing residuals, and proposing follow-up measurements—mirroring actual scientific practice. The 120 tasks systematically vary complexity by signal-to-noise ratio, number of planets, and orbital parameters, allowing fine-grained analysis of agent capabilities across difficulty tiers. The inclusion of real exoplanet datasets grounds the evaluation in authentic scientific challenges rather than synthetic toy problems, exposing how agents fail at constraint satisfaction despite achieving numerical optimization.
How It Works
Stargazer presents agents with radial-velocity time series—measurements of a star's motion caused by orbiting planets—and asks them to infer the orbital parameters (mass, period, eccentricity) of unseen exoplanets. The input is a time series of noisy velocity measurements; the agent must select actions like fitting Keplerian models, proposing additional observations, or adjusting priors on orbital parameters. The environment provides feedback through residual analysis and model comparison metrics, allowing agents to iteratively refine their fits. Tasks are stratified by signal-to-noise ratio (high SNR = obvious planets, low SNR = subtle signals) and multiplicity (1 to 3+ planets), with real archival data from exoplanet surveys embedded in 20 of the 120 problems. The agent's goal is to converge on physically plausible orbital solutions that satisfy Kepler's laws and stability constraints—not just minimize loss on held-out test data.
Production Impact
For engineers building autonomous research systems, Stargazer demonstrates that standard ML evaluation misses critical failure modes: agents may achieve good numerical metrics while violating domain constraints that domain experts consider non-negotiable. If you're building an AI system for scientific discovery, you must evaluate against task-specific constraints (energy conservation, stability conditions, measurement limits) rather than relying solely on loss curves or accuracy metrics. This changes your evaluation pipeline significantly—you'd need domain experts to define constraint checkers, build environments that enforce physical plausibility, and log not just agent accuracy but also constraint violation rates. The benchmark suggests you'll need hybrid architectures where agents can query physics simulators, receive constraint-guided feedback, and perform iterative refinement—not one-shot prediction. The production trade-off is clear: constraint-respecting systems require more compute (multi-step iteration, simulator calls) but catch errors that unconstrained agents would silently propagate into invalid scientific claims.
Limitations and When Not to Use This
Stargazer focuses narrowly on exoplanet detection via radial velocity, which limits generalization insights to other physics domains (particle physics, material science, climate modeling) where constraint vocabularies and feedback mechanisms differ substantially. The benchmark assumes agents have access to well-defined action spaces (fitting routines, observation strategies) that may not exist for less-structured domains; it's unclear how the framework scales when domain experts cannot enumerate valid actions a priori. With 120 tasks and only 20 real cases, the evaluation may not capture the long tail of edge cases or adversarial observational scenarios that appear in decades-long monitoring campaigns. The paper doesn't address computational cost—iterative refinement with physics simulation callbacks could be prohibitively expensive for large-scale agent deployment, and latency constraints in real survey pipelines aren't discussed.
Research Context
This work builds on two research streams: agent benchmarking (MuZero, BIG-Bench, GPQA) that moved beyond standard ML metrics, and scientific discovery systems (AutoML, neural architecture search, inverse design) that started incorporating domain knowledge. Stargazer advances the agent evaluation frontier by insisting that benchmarks must embed authentic scientific workflows—iterative hypothesis testing, constraint satisfaction, real observational noise—rather than static supervised learning setups. The exoplanet domain is ideal for this because it has ground truth (confirmed planets in archival surveys), high stakes (false positives waste telescope time), and clear physics-based constraints (Kepler's laws). This opens a research direction: building benchmarks for other sciences (protein folding validation, materials discovery, genomics) that similarly require agents to respect domain constraints while optimizing under uncertainty.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::