Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
| Authors | Chenyu Zhou et al. |
| Year | 2026 |
| HF Upvotes | 44 |
| arXiv | 2604.08224 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large language model (LLM) agents are increasingly built less by changing model weights than by reorganizing the runtime around them. Capabilities that earlier systems expected the model to recover internally are now externalized into memory stores, reusable skills, interaction protocols, and the surrounding harness that makes these modules reliable in practice. This paper reviews that shift through the lens of externalization. Drawing on the idea of cognitive artifacts, we argue that agent infrastructure matters not merely because it adds auxiliary components, but because it transforms hard cognitive burdens into forms that the model can solve more reliably. Under this view, memory externalizes state across time, skills externalize procedural expertise, protocols externalize interaction structure, and harness engineering serves as the unification layer that coordinates them into governed execution. We trace a historical progression from weights to context to harness, analyze memory, skills, and protocols as three distinct but coupled forms of externalization, and examine how they interact inside a larger agent system. We further discuss the trade-off between parametric and externalized capability, identify emerging directions such as self-evolving harnesses and shared agent infrastructure, and discuss open challenges in evaluation, governance, and the long-term co-evolution of models and external infrastructure. The result is a systems-level framework for explaining why practical agent progress increasingly depends not only on stronger models, but on better external cognitive infrastructure.
Engineering Breakdown
Plain English
This paper argues that modern LLM agents are built primarily through infrastructure engineering rather than model weight modification. Instead of training models to internalize capabilities like memory management, planning, and tool use, contemporary systems externalize these functions into dedicated components—memory systems, skill libraries, interaction protocols, and runtime harnesses. The authors frame this shift through cognitive science concepts of cognitive artifacts, showing that external structure transforms difficult reasoning problems into forms that models can solve more reliably. The key insight is that agent capabilities emerge from how the model interacts with its surrounding ecosystem, not just from internal model parameters.
Core Technical Contribution
The paper's core contribution is a conceptual framework that explains the paradigm shift from end-to-end learned models to modular, externalized agent architectures. Rather than treating auxiliary components as mere add-ons, the authors demonstrate that externalization fundamentally restructures the cognitive task landscape—moving from "model must learn everything internally" to "model works with specialized external systems." This reframes how we should evaluate agent capabilities: instead of measuring what's in model weights, we measure what emerges from the model-plus-infrastructure system. The framework draws explicitly on cognitive science literature to provide theoretical grounding for why this architectural split improves reliability and capability.
How It Works
The paper conceptualizes agent systems as composed of three primary externalizable components: memory (which externalizes state management across time), skills (which externalize procedural expertise as callable operations), and protocols (which externalize interaction rules and communication patterns). In this architecture, the LLM acts as a reasoning engine that orchestrates interactions with these components rather than trying to implement all logic internally. When an agent needs to solve a task, it decomposes the problem into calls to external systems—retrieving relevant context from memory, invoking specialized skills for subtasks, and following protocols that enforce valid interaction sequences. The runtime harness coordinates these components, handles error cases, manages state consistency, and ensures the model's outputs are valid and executable. This is mechanically simpler than training the model to do everything: instead of gradient descent on model parameters, you engineer interfaces between components.
Production Impact
For production systems, this framework shifts engineering effort from model training to infrastructure architecture. Teams building agents should invest in robust memory systems (vector databases, knowledge graphs, structured stores) rather than expecting fine-tuning to improve memory capacity. Similarly, encapsulating domain expertise as callable skills—via function libraries, external APIs, or specialized models—becomes more cost-effective than creating monolithic models. This reduces training costs significantly: you iterate on skill definitions and protocols without retraining, and you can swap components independently. However, there are trade-offs: additional I/O latency from external calls, increased operational complexity (managing memory consistency, skill versioning), and the need to maintain clear interfaces between components. For latency-critical applications, the overhead of external system calls may require careful optimization through batching, caching, and asynchronous design.
Limitations and When Not to Use This
The paper does not address how to optimally partition capabilities between internal model computation and external systems—this remains an empirical engineering question without principled guidelines. It assumes that external systems (memory stores, skill libraries, protocols) are well-engineered and reliable, but doesn't deeply explore failure modes when these components degrade or conflict. The framework also doesn't quantify when externalization helps versus hurts: for some tasks, training the model to internalize knowledge may be more efficient than maintaining complex external infrastructure. Additionally, the paper's reliance on cognitive science analogies provides intuition but not formal guarantees about when this architectural pattern improves reliability. Follow-up work is needed on: (1) automated partitioning strategies, (2) robustness to component failures, (3) principled measurement of when externalization is worth the engineering cost.
Research Context
This paper builds on decades of research in cognitive science on externalization and cognitive artifacts (Hutchins, Kirsh & Maglio), connecting those ideas to modern LLM systems. It fits within the broader move in AI systems away from end-to-end learning—similar to how NLP evolved from pure neural approaches to hybrid symbolic-neural systems with external knowledge bases. The work aligns with concurrent research on agent frameworks (like ReAct, AutoGPT, LangChain), tool use in LLMs, and retrieval-augmented generation (RAG). It also relates to research on model interpretability and mechanistic understanding: by externalizing capabilities, you make agent behavior more transparent and debuggable. The paper opens research directions into formalized agent design patterns, optimal capability partitioning, and how to verify correctness in externalized systems.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::