FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
:::info Stub — Full Engineering Breakdown Coming This paper has a linked code implementation and was featured on Hugging Face Papers with 17 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Runyuan He et al. |
| Year | 2026 |
| HF Upvotes | 17 |
| arXiv | 2605.14445 |
| Download | |
| Code | https://github.com/FrontierCS/FrontierSmith |
Abstract
Many real-world coding challenges are open-ended and admit no known optimal solution. Yet, recent progress in LLM coding has focused on well-defined tasks such as feature implementation, bug fixing, and competitive programming. Open-ended coding remains a weak spot for LLMs, largely because open-ended training problems are scarce and expensive to construct. Our goal is to synthesize open-ended coding problems at scale to train stronger LLM coders. We introduce FrontierSmith, an automated system for iteratively evolving open-ended problems from existing closed-ended coding tasks. Starting from competitive programming problems, FrontierSmith generates candidate open-ended variants by changing the problems'goals, restricting outputs, and generalizing inputs. It then uses a quantitative idea divergence metric to select problems that elicit genuinely diverse approaches from different solvers. Agents then generate test cases and verifiers for the surviving candidates. On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models: Qwen3.5-9B improves by +8.82 score on FrontierCS and +306.36 (Elo-rating-based performance) on ALE-bench; Qwen3.5-27B improves by +12.12 and +309.12, respectively. The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones, suggesting that closed-ended seeds can be a practical starting point for long-horizon coding data.
Engineering Breakdown
Plain English
FrontierSmith is an automated system that generates open-ended coding problems at scale by systematically transforming closed-ended competitive programming problems into variants with fuzzy goals, restricted outputs, or generalized inputs. The core finding is that you can use iterative evolution and quantitative filtering to create high-quality open-ended training data, addressing a major bottleneck in training LLMs for real-world coding tasks where problems don't have single correct answers.
Key Engineering Insight
The paper's key insight is that open-ended problem synthesis can be automated through structured transformations (goal modification, output restriction, input generalization) rather than manual construction, making it feasible to scale training data for realistic coding scenarios that LLMs currently struggle with.
Why It Matters for Engineers
Production LLM coding systems today are trained mostly on well-defined problems with clear specs and correctness criteria, but real engineering work is messy—requirements evolve, trade-offs exist, and there are multiple valid solutions. This paper directly addresses why your LLM-powered code generation system likely struggles with ambiguous requirements: there's almost no training data for it. Synthetic open-ended problems could significantly improve model performance on actual engineering tasks.
Research Context
Previous work focused LLM coding training on competitive programming, bug fixes, and feature implementation—all closed-ended tasks with definitive solutions. This paper shifts the paradigm by automating the creation of open-ended variants, filling a gap that existed because manually authoring such problems is expensive and subjective. This enables a new scale of training data and should improve LLM performance on the kinds of underspecified, ambiguous problems that dominate real software development.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::