Structured Distillation for Personalized Agent Memory: 11x Token Reduction with Retrieval Preservation
| Authors | Sydney Lewis |
| Year | 2026 |
| Field | AI / Agents |
| arXiv | 2603.13017 |
| Download | |
| Categories | cs.AI, cs.CL, cs.IR |
Abstract
Long conversations with an AI agent create a simple problem for one user: the history is useful, but carrying it verbatim is expensive. We study personalized agent memory: one user's conversation history with an agent, distilled into a compact retrieval layer for later search. Each exchange is compressed into a compound object with four fields (exchange_core, specific_context, thematic room_assignments, and regex-extracted files_touched). The searchable distilled text averages 38 tokens per exchange. Applied to 4,182 conversations (14,340 exchanges) from 6 software engineering projects, the method reduces average exchange length from 371 to 38 tokens, yielding 11x compression. We evaluate whether personalized recall survives that compression using 201 recall-oriented queries, 107 configurations spanning 5 pure and 5 cross-layer search modes, and 5 LLM graders (214,519 consensus-graded query-result pairs). The best pure distilled configuration reaches 96% of the best verbatim MRR (0.717 vs 0.745). Results are mechanism-dependent. All 20 vector search configurations remain non-significant after Bonferroni correction, while all 20 BM25 configurations degrade significantly (effect sizes |d|=0.031-0.756). The best cross-layer setup slightly exceeds the best pure verbatim baseline (MRR 0.759). Structured distillation compresses single-user agent memory without uniformly sacrificing retrieval quality. At 1/11 the context cost, thousands of exchanges fit within a single prompt while the verbatim source remains available for drill-down. We release the implementation and analysis pipeline as open-source software.
Engineering Breakdown
Plain English
This paper addresses a fundamental problem in long-running AI agent systems: conversation history becomes too expensive to store and search in full. The authors propose a personalized memory system that compresses each user-agent exchange into a compact, searchable object with four structured fields, reducing the average exchange from 371 tokens to 38 tokens—an 11x compression ratio. Testing on 4,182 conversations (14,340 exchanges) from software engineering projects, they demonstrate that this compression maintains recall quality on 201 real user queries, showing that aggressive distillation doesn't destroy the information users actually need to retrieve later.
Core Technical Contribution
The core novelty is a four-field compression schema for conversation memory that trades verbatim history for structured searchability. Instead of storing full exchanges, each interaction is distilled into: exchange_core (the essential interaction), specific_context (relevant details), thematic_room_assignments (topic/domain tagging), and regex-extracted_files_touched (structured metadata extraction). This approach is distinct from prior memory work because it treats compression not as lossy summarization but as semantic restructuring—extracting and re-organizing information for the specific retrieval patterns humans actually use. The insight is that users rarely need full context; they need precise pointers to relevant past interactions, which can be indexed and searched much more efficiently than raw conversation.
How It Works
The system processes each user-agent exchange through a four-stage distillation pipeline. First, exchange_core extracts the minimal decision or fact that was established (typically a few tokens). Second, specific_context captures domain-relevant details—variable names, error messages, API calls—that won't appear in natural language form again. Third, thematic_room_assignments labels the semantic domain or intent (e.g., 'debugging', 'refactoring', 'documentation') to cluster related exchanges. Fourth, regex-extracted_files_touched uses pattern matching to identify and normalize file paths, functions, or identifiers mentioned. These four fields are concatenated into a searchable index averaging 38 tokens per exchange. When a user queries (e.g., 'what did we discuss about authentication?'), the system searches across all four fields—matching both semantic tags and concrete artifacts—and retrieves the ranked exchanges, reconstructing full context on-demand only if needed.
Production Impact
In production, this approach directly reduces memory infrastructure cost by 11x while maintaining user experience. For a service managing millions of user conversations, this means storing weeks or months of history in the same footprint previously needed for days. Retrieval latency also improves: searching a 38-token index is orders of magnitude faster than semantic search over full exchanges. The structured fields enable hybrid retrieval—combining keyword search (files_touched), semantic matching (thematic_room_assignments), and metadata filters (specific_context)—without requiring vector embeddings or dense retrieval models. The tradeoff is reconstruction cost: when users need context beyond the distilled version, you must either store pointers to the original exchange or accept some information loss. Integration is straightforward—this sits as a preprocessing layer between raw conversation logs and the retrieval system, requiring minimal changes to existing agent architectures.
Limitations and When Not to Use This
The approach assumes that meaningful compression can be extracted via rule-based extraction (regex for files) and semantic tagging, which may fail for novel or ambiguous domains. The four-field schema is domain-specific—optimized for software engineering conversations—and may not generalize to medical, legal, or creative domains where context preservation is non-negotiable. The paper's evaluation is incomplete (the abstract cuts off mid-evaluation), so it's unclear how recall performance degrades on adversarial or rare queries, or how well this works for non-English or non-Latin-script languages. There's also an open question about drift: if a user's project structure evolves (files renamed, refactored), do file extraction tags remain valid, or do you need periodic re-indexing?
Research Context
This work sits at the intersection of two active research areas: efficient long-context modeling (following work on sparse attention, retrieval-augmented generation, and prompt compression) and personalized AI systems (building on memory networks and user-specific adaptation). It contrasts with recent approaches that use learned compression (e.g., LLM-based summarization or neural quantization) by showing that structured, rule-based distillation can be competitive and more interpretable. The contribution opens a research direction on hybrid retrieval—combining symbolic (regex, tags) and semantic (thematic assignments) signals—which is gaining traction in retrieval-augmented LLM systems and knowledge management.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::