Combee: Scaling Prompt Learning for Self-Improving Language Model Agents
| Authors | Hanchen Li et al. |
| Year | 2026 |
| HF Upvotes | 26 |
| arXiv | 2604.04247 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Recent advances in prompt learning allow large language model agents to acquire task-relevant knowledge from inference-time context without parameter changes. For example, existing methods (like ACE or GEPA) can learn system prompts to improve accuracy based on previous agent runs. However, these methods primarily focus on single-agent or low-parallelism settings. This fundamentally limits their ability to efficiently learn from a large set of collected agentic traces. It would be efficient and beneficial to run prompt learning in parallel to accommodate the growing trend of learning from many agentic traces or parallel agent executions. Yet without a principled strategy for scaling, current methods suffer from quality degradation with high parallelism. To improve both the efficiency and quality of prompt learning, we propose Combee, a novel framework to scale parallel prompt learning for self-improving agents. Combee speeds up learning and enables running many agents in parallel while learning from their aggregate traces without quality degradation. To achieve this, Combee leverages parallel scans and employs an augmented shuffle mechanism; Combee also introduces a dynamic batch size controller to balance quality and delay. Evaluations on AppWorld, Terminal-Bench, Formula, and FiNER demonstrate that Combee achieves up to 17x speedup over previous methods with comparable or better accuracy and equivalent cost.
Engineering Breakdown
Plain English
Combee addresses the challenge of scaling prompt learning for LLM-based agents when operating in high-parallelism settings. Existing methods like ACE and GEPA can learn system prompts from agent execution traces to improve task performance, but they degrade in quality when parallelized across multiple agent runs. The paper proposes a principled strategy for scaling prompt learning that maintains quality while leveraging parallel agent executions and large collections of agentic traces, solving a critical bottleneck for production systems that need to learn from massive amounts of interaction data efficiently.
Core Technical Contribution
The core innovation is a parallelization strategy for prompt learning that prevents quality degradation when scaling from single-agent to high-parallelism settings. Rather than naively parallelizing existing prompt optimization methods, the authors introduce a principled approach that maintains learning signal integrity across parallel agent runs, likely through aggregation mechanisms or distributed gradient computation that preserves the optimization landscape. This enables prompt learning systems to process many agentic traces simultaneously without the typical trade-off between throughput and final prompt quality that plague naive scaling approaches.
How It Works
The system operates by collecting execution traces from multiple parallel LLM agent runs on a task, where each trace includes input, reasoning steps, outputs, and feedback signals. These traces are fed into the prompt learning optimizer, which uses a scaling-aware aggregation strategy—likely involving coordinated updates across parallel workers or hierarchical gradient collection—to synthesize improved system prompts without convergence degradation. The learned prompts are then deployed to the next batch of agents, creating a closed loop where parallel execution both produces training signal and benefits from improved prompts. The key technical mechanism appears to involve careful synchronization and weighting of information from multiple parallel streams so that no single worker's noisy gradient dominates the optimization.
Production Impact
For production systems, Combee enables real-time improvement of agentic behavior without retraining model weights, which is critical for systems deployed at scale. Teams can run thousands of agent instances in parallel collecting traces, then simultaneously learn improved prompts that get pushed to all subsequent instances—creating a virtuous cycle of rapid improvement. The approach reduces time-to-improvement from days (batch retraining) to hours, and eliminates the need for parameter-efficient fine-tuning infrastructure. Trade-offs include added complexity in distributed prompt optimization (requiring careful synchronization logic), potential staleness if aggregate learning lags behind real-time agent deployment, and the need for robust feedback mechanisms on each agent trace to properly weight learning signals.
Limitations and When Not to Use This
The paper assumes reliable feedback signals are available for every agent trace, which is unrealistic when human annotation is required—the method degrades gracefully with noisy or sparse labels but this isn't thoroughly analyzed. It focuses on system prompt learning and doesn't address learning of in-context examples or few-shot demonstrations, which may be equally important for performance. The approach is likely constrained to relatively small prompt spaces (hundreds to thousands of tokens) and may not extend to learning larger retrieval-augmented or tool-use components. Additionally, there's tension between parallelism degree and prompt stability—at extreme parallelism, the aggregated prompt signal may become inconsistent or oscillate, requiring careful tuning of aggregation hyperparameters that may be task-dependent.
Research Context
This work extends recent prompt learning methods like ACE (Agentic Computing Engine) and GEPA that demonstrated in-context learning without parameter updates, building on the insight that LLMs can acquire task knowledge through system prompt optimization. It addresses a concrete pain point in the scaling pipeline identified by the broader agentic AI community working on self-improving systems. The research connects to distributed optimization literature (federated learning, multi-worker SGD) and reinforces the trend toward learning from interaction at inference time rather than pre-training. This opens future research into adaptive parallelism strategies, online prompt consolidation, and failure recovery in distributed agentic learning loops.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::