From Context to Skills: Can Language Models Learn from Context Skillfully?
:::info Stub — Full Engineering Breakdown Coming This paper was featured on Hugging Face Daily Papers on 2026-05-03 with 149 upvotes. A full breakdown with production viability rating, implementation notes, and honest limitations is being written. Subscribe to AI Letters → :::
| Authors | Shuzheng Si et al. |
| Year | 2026 |
| HF Upvotes | 149 |
| arXiv | 2604.27660 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Many real-world tasks require language models (LMs) to reason over complex contexts that exceed their parametric knowledge. This calls for context learning, where LMs directly learn relevant knowledge from the given context. An intuitive solution is inference-time skill augmentation: extracting the rules and procedures from context into natural-language skills. However, constructing such skills for context learning scenarios faces two challenges: the prohibitive cost of manual skill annotation for long, technically dense contexts, and the lack of external feedback for automated skill construction. In this paper, we propose Ctx2Skill, a self-evolving framework that autonomously discovers, refines, and selects context-specific skills without human supervision or external feedback. At its core, a multi-agent self-play loop has a Challenger that generates probing tasks and rubrics, a Reasoner that attempts to solve them guided by an evolving skill set, and a neutral Judge that provides binary feedback. Crucially, both the Challenger and the Reasoner evolve through accumulated skills: dedicated Proposer and Generator agents analyze failure cases and synthesize them into targeted skill updates for both sides, enabling automated skill discovery and refinement. To prevent adversarial collapse caused by increasingly extreme task generation and over-specialized skill accumulation, we further introduce a Cross-time Replay mechanism that identifies the skill set achieving the best balance across representative cases for the Reasoner side, ensuring robust and generalizable skill evolution. The resulting skills can be plugged into any language model to obtain better context learning capability. Evaluated on four context learning tasks from CL-bench, Ctx2Skill consistently improves solving rates across backbone models.
Engineering Breakdown
Plain English
This paper addresses a real problem: LLMs often need to reason over contexts (documents, code, data) that contain information beyond their training data, but manually creating skills or rules from those contexts is expensive and impractical at scale. The authors propose Ctx2Skill, an automated framework that discovers, refines, and selects context-specific skills without human annotation or external feedback signals, essentially teaching models to extract and apply procedural knowledge from long, complex contexts on their own.
Key Engineering Insight
The core innovation is self-supervised skill evolution—the system bootstraps its own training signal by discovering skills in context, using the model's own predictions to evaluate whether those skills are useful, then iterating without human labeling. This breaks the typical bottleneck where in-context learning requires either expensive annotation or external evaluation oracles.
Why It Matters for Engineers
Production systems constantly deal with domain-specific documents, APIs, and technical specifications that change faster than model retraining cycles. Right now, teams either manually prompt-engineer skills or accept degraded performance on out-of-distribution contexts. Automated skill discovery at inference time could let deployed models adapt to new procedural knowledge in customer data without retraining, reducing the gap between static parametric knowledge and dynamic real-world contexts.
Research Context
Prior work on in-context learning focused on few-shot examples or chain-of-thought prompting, but these assume human-crafted demonstrations. This paper advances the autonomy side—moving from 'LMs learn from context humans provide' to 'LMs autonomously extract and refine their own skills from context.' It sits between retrieval-augmented generation (which pulls facts) and finetuning (which modifies weights), offering a middle ground for procedural adaptation at runtime.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::