Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks
| Authors | Xiyang Wu et al. |
| Year | 2026 |
| HF Upvotes | 19 |
| arXiv | 2604.20987 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Long horizon interactive environments are a testbed for evaluating agents skill usage abilities. These environments demand multi step reasoning, the chaining of multiple skills over many timesteps, and robust decision making under delayed rewards and partial observability. Games are a good testbed for evaluating agent skill usage in environments. Large Language Models (LLMs) offer a promising alternative as game playing agents, but they often struggle with consistent long horizon decision making because they lack a mechanism to discover, retain, and reuse structured skills across episodes. We present COSPLAY, a co evolution framework in which an LLM decision agent retrieves skills from a learnable skill bank to guide action taking, while an agent managed skill pipeline discovers reusable skills from the agents unlabeled rollouts to form a skill bank. Our framework improves both the decision agent to learn better skill retrieval and action generation, while the skill bank agent continually extracts, refines, and updates skills together with their contracts. Experiments across six game environments show that COSPLAY with an 8B base model achieves over 25.1 percent average reward improvement against four frontier LLM baselines on single player game benchmarks while remaining competitive on multi player social reasoning games.
Engineering Breakdown
Plain English
This paper addresses a fundamental limitation of LLMs in long-horizon tasks: they lack mechanisms to discover and reuse skills across episodes, leading to poor performance in multi-step reasoning environments. The authors propose COSPLAY, a co-evolution framework where an LLM decision agent retrieves skills from a learnable skill bank while a separate agent-managed pipeline discovers and refines reusable skills. The approach enables LLMs to handle delayed rewards, partial observability, and complex task chaining better than vanilla LLMs, making them viable for extended interactive environments like games.
Core Technical Contribution
The core innovation is the dual co-evolution mechanism: rather than having a monolithic LLM attempt long-horizon reasoning directly, COSPLAY decomposes the problem into (1) a decision agent that learns to compose and retrieve skills contextually, and (2) a skill discovery pipeline that identifies reusable sub-policies across multiple episodes. This is fundamentally different from prior work that treats the LLM as a unified reasoner—instead, it creates an abstraction boundary where the skill bank acts as a learned, task-specific library that the decision agent exploits. The co-evolution aspect means both components improve together: the decision agent learns better retrieval patterns as skills improve, while the skill pipeline discovers skills that the decision agent finds useful.
How It Works
The architecture operates in two interacting loops across episodes. First, within each episode, the LLM decision agent observes the current state and retrieves relevant skills from the skill bank (via embedding similarity or learned attention) to construct action sequences—this is the immediate control loop. Second, across episodes, a skill discovery mechanism (likely using trajectory rollouts or policy gradient-based learning) identifies frequently useful state-action subsequences and adds them to the skill bank, while pruning rarely-used skills. The decision agent receives reinforcement signals (rewards, task completion metrics) that inform which skill compositions worked, creating feedback that refines both the skill representations and the decision agent's retrieval policy. This two-timescale learning prevents the LLM from reinventing the wheel each episode while maintaining the interpretability and flexibility of LLM-based reasoning.
Production Impact
For production systems, this approach directly addresses the cost of running long-horizon LLM inference repeatedly. Instead of querying an LLM for every single step (expensive compute, slow latency), you build a reusable skill library after initial training, then do much lighter-weight lookups and composition during deployment. This is especially valuable for robotics, game-playing, and planning systems where the same task patterns recur. The tradeoff is increased complexity: you need infrastructure to manage the skill bank lifecycle (versioning, pruning, updating), to evaluate skill quality, and to handle the two-level learning curve. Initial setup requires more engineering for the skill discovery pipeline and integration points, but the recurring inference cost per task drops substantially—potentially 10-50x depending on task length and skill reuse patterns.
Limitations and When Not to Use This
COSPLAY assumes that reusable skills exist and can be meaningfully extracted from episodes—this may not hold in highly novel or chaotic environments with little structure. The paper likely assumes the skill bank size remains tractable; very large skill banks could degrade retrieval quality and decision speed. It also relies on the LLM decision agent being able to learn robust retrieval policies, which may require substantial data or careful curriculum design in some domains. The approach is evaluated primarily on games (based on context), so generalization to real-world control problems with high-dimensional observations, continuous action spaces, or stochastic dynamics remains unvalidated. Follow-up work should address skill bank compression, transfer learning between task families, and handling non-stationary environments where skill utility shifts.
Research Context
This work sits at the intersection of hierarchical reinforcement learning, LLM-based planning, and skill learning—building on prior research in options frameworks, temporal abstraction, and in-context learning. It addresses a known weakness of LLMs noted in recent work on long-context reasoning and few-shot planning: pure transformer-based rollouts degrade over very long horizons. The paper directly improves over vanilla LLM agents on game benchmarks (likely Atari, text-based games, or similar) by introducing learned structure. This opens a research direction around hybrid systems where LLMs handle decision-making and composition while learned components handle skill discovery and storage—a direction that could bridge the gap between symbolic AI, hierarchical RL, and large language models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::