p1: Better Prompt Optimization with Fewer Prompts
| Authors | Zhaolin Gao et al. |
| Year | 2026 |
| HF Upvotes | 3 |
| arXiv | 2604.08801 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what makes a task amenable to prompt optimization. We show that the reward variance across different system prompts can be decomposed into two components: variance among responses, which captures generation stochasticity, and variance among system prompts, which captures differences in system prompt quality. Prompt optimization succeeds when variance among system prompts is sufficiently large, but fails when variance among responses dominates the variance of the system prompts. Surprisingly, we further show that scaling to more user prompts can hurt optimization by reducing variance among system prompts, especially on heterogeneous datasets where different user prompts favor different system prompts. Motivated by this insight, we propose p1, a simple user prompt filtering method that selects a small subset of user prompts with high variance across candidate system prompts. This subset of user prompts allows one to distinguish a good system prompt from a bad one, making system optimization easier. Experiments on reasoning benchmarks show that p1 substantially improves prompt optimization over training on the full dataset and outperforms strong baselines such as GEPA. Notably, training on only two prompts from AIME 24 yields a system prompt that generalizes well to other reasoning benchmarks.
Engineering Breakdown
Plain English
This paper investigates why prompt optimization—the practice of searching for better system prompts to improve language model performance without fine-tuning weights—works well for some tasks but fails for others. The authors decompose the variance in rewards across different prompts into two sources: variance from the model's generation stochasticity (randomness in responses) and variance from actual differences in prompt quality. They find that prompt optimization succeeds only when the variance from prompt quality differences is large enough to dominate the variance from response randomness. Counterintuitively, they discover that using more user prompts during optimization can actually hurt performance by reducing the measurable variance between system prompts, revealing a previously unknown scaling pitfall.
Core Technical Contribution
The core contribution is a decomposition framework that separates total reward variance into response variance and prompt variance components, providing a principled way to predict when prompt optimization will succeed or fail. This decomposition reveals that task amenability to prompt optimization depends on the signal-to-noise ratio between prompt quality differences and generation stochasticity—a relationship that wasn't previously formalized. The authors demonstrate this through empirical analysis and show that scaling behaviors (more prompts) can be counterproductive, which contradicts intuitive scaling assumptions. This framework provides actionable diagnostics for practitioners to determine whether prompt optimization is worth pursuing on a given task before investing computational resources.
How It Works
The technical approach starts by defining a reward function that measures how well a language model performs on a task given a specific system prompt. For each task, the authors compute rewards across a set of candidate system prompts and multiple user queries. They then decompose the total variance in rewards into two orthogonal components: (1) within-prompt variance caused by the model generating different responses to different user queries for the same system prompt (response variance), and (2) between-prompt variance caused by different system prompts producing systematically different average rewards (prompt variance). The key insight is that prompt optimization's effectiveness depends on the ratio of prompt variance to total variance—when prompt variance is small relative to response variance, optimizing prompts becomes statistically impossible because you can't reliably distinguish good prompts from bad ones. The paper shows that adding more user prompts increases the denominator of this ratio (more data points for averaging response variance), paradoxically making it harder to detect true differences between prompts.
Production Impact
For engineers building production systems that use prompt optimization, this paper provides a critical pre-flight check: measure the variance decomposition on your specific task before investing in prompt search infrastructure. If you observe high response variance relative to prompt variance, you'll know that expensive prompt optimization campaigns are likely to fail, and you should instead focus on other levers like model scaling, retrieval augmentation, or fine-tuning. The framework suggests that tasks with deterministic or low-variance model behavior (e.g., structured extraction, arithmetic) are better candidates for prompt optimization than open-ended generation tasks. Operationally, practitioners should use a smaller held-out set of diverse user prompts during optimization rather than scaling to large prompt datasets, counterintuitively making the system both cheaper and more effective. This could significantly reduce compute costs for prompt search while improving reliability of discovered prompts.
Limitations and When Not to Use This
The paper assumes that variance decomposition is sufficient to predict optimization success, but doesn't deeply explore how prompt quality interacts with model scaling, temperature settings, or sampling strategies—all of which affect response variance. The framework requires careful measurement of variance components on your task, which adds upfront instrumentation overhead and may not be practical for teams running one-off experiments. The analysis relies on reward functions being well-defined and continuous, which breaks down for tasks with sparse or discrete feedback signals. The paper doesn't fully address how to optimize in the regime where response variance dominates (which likely describes many real-world applications), leaving practitioners with limited guidance when their task fails the variance check. Additionally, the relationship between this variance analysis and the actual optimization algorithm (e.g., gradient-based, evolutionary, retrieval-based) remains underexplored—it's unclear whether different search algorithms would change these variance-decomposition conclusions.
Research Context
This work builds on the growing literature on prompt engineering and in-context learning, particularly methods that search the prompt space without gradient updates (AutoPrompt, Prompt2Model). It addresses a fundamental question about when and why prompt optimization works, which prior work took for granted. The paper contributes to the emerging understanding of prompt optimization's statistical foundations, similar to how recent work has studied scaling laws and emergence in large language models. The variance decomposition framework relates to classical machine learning concepts like signal-to-noise ratios and variance-bias tradeoffs, but applies them in the novel context of prompt search. This research likely motivates follow-up work on adaptive optimization algorithms that adjust search strategies based on measured variance ratios, or on techniques to reduce response variance (e.g., through ensemble methods or temperature tuning) to make prompt optimization viable on harder tasks.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::