SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
| Authors | Shanshan Zhong et al. |
| Year | 2026 |
| HF Upvotes | 14 |
| arXiv | 2604.20087 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.
Engineering Breakdown
Plain English
This paper introduces SkillLearnBench, the first systematic benchmark for evaluating how LLM agents can automatically learn and acquire new skills through continual learning. The benchmark contains 20 real-world tasks across 15 sub-domains, grounded in an actual skill taxonomy, and evaluates skill quality at three levels: the skill itself, the execution trajectory, and final task outcome. The authors tested multiple continual learning approaches—including one-shot learning, self/teacher feedback mechanisms, and skill creator systems that generate skills from agent experiences—and found that all improved over baselines without skills, though consistent gains across all methods remained unclear. This work addresses a critical gap: while LLM agents have become powerful with skills, nobody had a rigorous way to measure or compare different skill-learning methods before.
Core Technical Contribution
The core contribution is SkillLearnBench itself—a rigorously designed, multi-level evaluation framework that moves beyond simple task success metrics to assess skill quality, execution quality, and outcome quality independently. Unlike prior work that evaluated skills in isolation or on toy domains, this benchmark uses a real-world skill taxonomy to ground 20 verified, skill-dependent tasks that genuinely require learned skills to solve. The authors also provide the first systematic comparison of four distinct skill-learning paradigms: continual learning methods, one-shot learning approaches, feedback-based methods (self and teacher), and skill creator systems that synthesize skills from agent rollouts. This three-pronged evaluation (skill quality × execution trajectory × task outcome) reveals nuances that single-metric benchmarks miss, enabling researchers to diagnose whether failures stem from poorly-learned skills, suboptimal execution, or task misalignment.
How It Works
The system works by taking an LLM agent and a skill-learning method, then iteratively building a skill library through agent experiences on real-world tasks. For each task, the agent attempts execution with its current skill set; the method then acquires new skills either by extracting reusable patterns from successful trajectories (continual learning), learning from a single demonstration (one-shot), receiving explicit feedback that refines skill definitions (teacher feedback), or having a separate skill creator module synthesize abstractions from experience batches. At evaluation time, three independent metrics assess the result: (1) skill quality checks whether learned skills are internally consistent and generalizable, (2) execution trajectory measures whether the agent follows a sensible action sequence with minimal backtracking or inefficiency, and (3) task outcome simply records success/failure. The benchmark runs this pipeline across 20 tasks spanning 15 sub-domains (drawn from a validated taxonomy), allowing researchers to pinpoint whether a method fails at skill acquisition, skill application, or task-level reasoning.
Production Impact
For teams building LLM agent systems, this benchmark provides the missing evaluation harness to decide which skill-learning strategy to deploy. In production, you typically have three choices: hand-write skills (expensive, non-adaptive), use continual learning to grow skills online (requires safe rollout and monitoring), or pre-train a skill library offline then freeze it (faster inference, lower variance). SkillLearnBench lets you measure the real cost of each approach on tasks similar to your domain—you can now quantify whether a one-shot learning method achieves 70% task success but poor execution efficiency, versus a teacher-feedback system that hits 65% success but with much cleaner trajectories (important if interpretability matters). The three-level evaluation also surfaces hidden problems: a method might achieve 80% task success because the underlying skill is correct but execution is inefficient, signaling that you should focus on a better planner rather than retraining skills. Integration is straightforward—plug in your skill learner and task executor, run against the benchmark, and get actionable diagnostics instead of a single pass/fail metric.
Limitations and When Not to Use This
The benchmark is grounded in a single real-world skill taxonomy, so the 20 tasks may not represent your specific domain distribution—tasks in autonomous driving, medical AI, or financial trading could behave very differently than the taxonomy used here. The paper does not provide statistical significance testing or confidence intervals for the results, making it unclear whether observed differences between methods are real or noise, especially given that early results were incomplete ('consistent gains remain elu[sive]' suggests the paper may be draft). The evaluation assumes that skill quality, execution trajectory, and task outcome can be measured independently, but in practice these metrics are often coupled—a poorly-learned skill might cause poor execution which causes task failure, conflating what should be separate signals. Additionally, all tested methods still underperformed or showed inconsistent wins, suggesting that the benchmark may be highlighting a fundamental limitation in current skill-learning approaches rather than enabling clear winners, and more research is needed to understand why continual learning methods plateau.
Research Context
This work builds on a decade of research in agent learning and skill discovery, but is the first to systematically benchmark skill acquisition at scale. It extends prior work on hierarchical reinforcement learning (which learned skill-like abstractions) and recent LLM agent papers (which used skills as prompts) by providing the rigorous evaluation protocol those fields lacked. The benchmark sits alongside other recent agent benchmarks like WebArena and SWE-Bench, but uniquely focuses on the skill-learning process itself rather than just end-task performance, opening a new research direction: can we design skill learners that consistently improve with scale? By grounding tasks in a real taxonomy, it also establishes a foundation for future work in cross-domain skill transfer and continual learning without catastrophic forgetting—two open problems in agent research.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::