How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings
| Authors | Yujian Liu et al. |
| Year | 2026 |
| HF Upvotes | 38 |
| arXiv | 2604.04323 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Agent skills, which are reusable, domain-specific knowledge artifacts, have become a popular mechanism for extending LLM-based agents, yet formally benchmarking skill usage performance remains scarce. Existing skill benchmarking efforts focus on overly idealized conditions, where LLMs are directly provided with hand-crafted, narrowly-tailored task-specific skills for each task, whereas in many realistic settings, the LLM agent may have to search for and select relevant skills on its own, and even the closest matching skills may not be well-tailored for the task. In this paper, we conduct the first comprehensive study of skill utility under progressively challenging realistic settings, where agents must retrieve skills from a large collection of 34k real-world skills and may not have access to any hand-curated skills. Our findings reveal that the benefits of skills are fragile: performance gains degrade consistently as settings become more realistic, with pass rates approaching no-skill baselines in the most challenging scenarios. To narrow this gap, we study skill refinement strategies, including query-specific and query-agnostic approaches, and we show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality. We further demonstrate the generality of retrieval and refinement on Terminal-Bench 2.0, where they improve the pass rate of Claude Opus 4.6 from 57.7% to 65.5%. Our results, consistent across multiple models, highlight both the promise and the current limitations of skills for LLM-based agents. Our code is available at https://github.com/UCSB-NLP-Chang/Skill-Usage.
Engineering Breakdown
Plain English
This paper addresses a critical gap in how we evaluate LLM-based agents using external skills—specifically, moving beyond unrealistic benchmarks where perfect, task-specific skills are handed to agents. Instead, the authors conduct the first large-scale study of skill utility in realistic conditions where agents must search and select from a collection of 34,000 real-world skills, and where available skills may not be well-aligned with the target task. The key finding is that existing benchmarks dramatically overestimate how well agents actually perform when they have to discover and adapt skills independently. By studying progressively harder settings, the paper establishes a foundation for understanding when agents can effectively leverage external knowledge versus when they struggle.
Core Technical Contribution
The core innovation is a comprehensive skill benchmarking framework that moves from idealized oracle conditions (perfect skills provided) to realistic retrieval-based conditions where agents must find relevant skills from a large, heterogeneous skill repository. The authors introduce a progression of experimental difficulty levels that isolates the challenges of skill discovery, selection, and adaptation separately, allowing researchers to measure the actual performance gap between what agents achieve with hand-crafted skills versus what they achieve in the wild. This methodology enables the first quantitative characterization of skill utility degradation as realism increases—a foundational measurement that prior work glossed over. The contribution is primarily empirical and methodological rather than algorithmic, but it exposes critical limitations in how we currently evaluate agent systems.
How It Works
The evaluation framework operates in stages of increasing difficulty: Stage 1 provides agents with oracle skills perfectly matched to each task; Stage 2 requires agents to retrieve skills from a large pool via semantic similarity matching; Stage 3 uses real-world skills that may be only loosely related to the target task; Stage 4 simulates scenarios where agents lack even approximately matching skills. For each stage, agents (implemented as LLM-based systems) receive a task description and either direct access to skills or a skill retrieval mechanism. The retrieval typically uses embedding-based search over a repository of 34,000 skills with metadata and descriptions. The agent then decides whether and how to use retrieved skills, adapting them to the task at hand. Success metrics measure task completion rate, and performance degradation across stages quantifies the practical utility loss when moving from idealized to realistic conditions.
Production Impact
For engineers building real agent systems, this paper provides concrete evidence that skill repository design and retrieval mechanisms are critical bottlenecks—not the skills themselves. In production, you cannot assume agents will receive curated, task-specific skills; you must invest in skill organization, semantic indexing, and potentially skill composition or adaptation layers. The 34k-skill benchmark establishes a realistic scale for skill repositories, suggesting production systems need robust retrieval pipelines (embedding models, re-ranking) to surface useful skills from noise. This impacts architecture: you may need hierarchical skill organization, skill summarization, or agent-mediated skill curation rather than assuming agents will correctly use any retrieved skill. The latency trade-off is non-trivial—semantic retrieval over 34k skills adds inference overhead, so you must consider whether to cache skill embeddings, use approximate nearest-neighbor search, or pre-filter by domain to maintain acceptable response times.
Limitations and When Not to Use This
The paper does not address how agents should adapt or compose retrieved skills to novel tasks—it measures whether agents choose and apply skills, but the underlying mechanism for skill customization is unclear. The benchmark assumes skills are standalone artifacts with clear descriptions; it may not capture the complexity of skills with hidden dependencies, conflicting interfaces, or non-obvious utility. The study is limited to LLM-based agents and does not explore whether reinforcement learning or fine-tuning could improve skill selection and adaptation over pure retrieval. Additionally, the 34k skills are real-world artifacts but likely biased toward common domains; the framework may not generalize to highly specialized skill repositories (robotics, scientific computing) where skill semantics are more technical and harder to retrieve via text embedding.
Research Context
This work builds on the growing trend of using modular skills and tools to augment LLMs—extending prior work on tool use (Toolformer, Code Interpreter) and retrieval-augmented generation to the domain of reusable, domain-specific knowledge. The paper complements concurrent research on agent planning and tool selection by isolating the skill retrieval bottleneck that most prior benchmarks (which provide oracle tools) have overlooked. It opens a new research direction: skill discovery and composition, where agents may need to combine multiple imperfect skills or learn to adapt skills on-the-fly. The 34k-skill benchmark will likely become a standard evaluation resource, similar to how datasets like HumanEval defined tool-use evaluation, enabling future work on skill embedding models, retrieval ranking, and multi-skill reasoning.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::