Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 12 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Jingjie Ning et al. |
| Year | 2026 |
| HF Upvotes | 12 |
| arXiv | 2605.05724 |
| Download | |
| Code | https://github.com/cxcscmu/Auto-Research-Recipes |
Abstract
We study auto research as a closed empirical loop driven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials. The central empirical finding is that lineage feedback lets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into later program-level recipe edits rather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by 0.81%, raises NanoChat-D12 CORE by 38.7%, and reduces CIFAR-10 Airbench96 wallclock by 4.59%, with each task measured by its own external evaluator and legality checks. The trace includes a strict architecture-domain audit of 157 headline-run submissions and program rewrites such as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.
Engineering Breakdown
Plain English
This paper describes an automated research system where specialist AI agents iteratively propose and test training recipe changes (hyperparameters, data processing, model architecture tweaks) in a closed loop. Instead of generating papers, the system produces an auditable trail of code diffs, experiments, and outcomes. Across nearly 1,200 trials, they found that agents trained on lineage feedback—learning from past failures like crashes, memory overruns, and accuracy misses—produce significantly better recipes than agents that treat each experiment independently.
Key Engineering Insight
The critical finding is that failure lineage matters: agents that understand why previous experiments failed (not just their scores) can propose smarter, targeted code edits in the next iteration, rather than making one-shot changes. This transforms failure data into actionable signal for systematic improvement.
Why It Matters for Engineers
For engineers shipping production models, this directly addresses the bottleneck of recipe engineering—the tedious, trial-and-error process of tuning training pipelines. Automating recipe search with failure-aware agents could reduce weeks of manual hyperparameter tuning and ablation studies, letting teams iterate faster and deploy better-tuned models without hand-crafted heuristics.
Research Context
Prior work on AutoML and hyperparameter optimization treated each trial as independent; this paper advances the state by making failures first-class data. It shows that structured feedback loops—where agents interpret crashes, budget violations, and numerical failures as program constraints—enables continuous refinement. This opens a path toward self-improving ML workflows that don't require human intervention to debug why an experiment failed.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::