Memory Compression and Summarization
The Agent That Forgot Everything
In January 2024, an engineering team at a fintech startup deployed what they called their "due diligence agent" - an AI system tasked with performing comprehensive research on acquisition targets. The work was substantial: reading SEC filings, analyzing financial statements, cross-referencing news articles, building competitive landscape models. A proper human analyst would spend two to three weeks on a single target. The team hoped the agent could compress this to two to three days.
It started well. On day one, the agent gathered company background information, read the founding story, identified key executives, and mapped the product portfolio. On day two, it began financial analysis: revenue trends, margin structure, customer concentration. By hour 36, something started going wrong. The agent began contradicting itself - noting in one paragraph that the company's gross margin was "industry-leading at 78%" and in the next paragraph describing the company as "struggling with margin pressure." When the team examined the logs, the cause was obvious. The context window had been filling up. Old memories were being pushed out. The agent had lost access to its day-one research when it needed it most.
This is not a rare failure mode - it is the most common failure mode for any agent that runs longer than an hour. Every language model operates within a finite context window. For Claude, that window is large (200K tokens), but even 200K tokens fills up faster than you might expect: a single large SEC filing is 50-100K tokens, detailed financial analysis generates another 30K tokens of intermediate reasoning, and conversation history accumulates relentlessly. Without active memory management, the agent reaches context capacity and the framework starts silently discarding old content. The agent is amnesiac by design.
The due diligence team's agent had no memory management strategy. Content fell out of context in the order it was added - oldest first - with no consideration of what was important. The company background (day one, added first) was gone before the team reached day two's analysis. The agent was doing detailed financial work without access to the strategic context that made the financial numbers meaningful.
Memory compression is the discipline of managing what an agent remembers as time passes and work accumulates. Done well, it enables agents to run for days without losing critical context. Done poorly - or not at all - it produces the amnesiac agent, confident in conclusions built on foundations it can no longer access.
:::tip 🎮 Interactive Playground Visualize this concept: Try the Context Compression demo on the EngineersOfAI Playground - no code required. :::
Why This Exists - The Context Window as a Finite Resource
The Raw Problem
A language model's context window is not a hard disk. It is more like a physical desk. You can only have so many papers spread out at once. When new papers arrive, old ones must go somewhere - either filed away (external memory), summarized into a briefing document, or thrown away. For most agent frameworks without explicit memory management, "thrown away" is the default.
The mathematical reality: at 200K token context windows and an average of 750 tokens per message (a typical agent turn), you get roughly 260 turns before the context is full. For an agent taking 30-second turns, that is 130 minutes of runtime before hitting context limits. For a due diligence agent working on a major acquisition, 130 minutes is barely the start.
But token count is only part of the problem. Even within the context window, attention quality degrades for information that is far from the current position. Research on the "lost in the middle" phenomenon (Liu et al., 2023) demonstrated that language models perform significantly worse at retrieving information placed in the middle of long contexts compared to the beginning or end. An agent whose important context sits 150K tokens back in a 200K token window may effectively have lost access to it anyway - the model's attention weights deprioritize distant content.
The Naive Solutions and Why They Fail
Truncation - simply cutting off the oldest content - is what most naive implementations do. It is fast and simple but disastrous for agents with multi-day tasks. The oldest content is often the most important: initial task briefings, key constraints, critical facts established early in the session. Truncation discards it first.
Sliding window - maintaining only the last N tokens - is marginally better but has the same fundamental problem. Important context from earlier in the session is lost as time passes.
Full storage in external memory - offloading everything to a vector database and retrieving relevant pieces on demand - avoids information loss but has serious retrieval problems. The agent must decide what to retrieve before it knows what it needs. Important context that does not match the current query stays unretriered. Retrieval latency adds up across many calls. And the agent still has a limited context window; it cannot retrieve everything at once.
Effective memory compression addresses the root issue: proactively identifying what is important, compressing the rest, and maintaining a structured hierarchy so that critical information is always accessible.
Historical Context - From Human PKM to Agent Memory
The intellectual foundation for agent memory compression comes from an unlikely source: personal knowledge management theory.
Tiago Forte popularized Progressive Summarization in his 2017 essay and subsequent book "Building a Second Brain" (2022). The key insight: when you read a note, highlight the most important sentences. When you revisit it, bold the most important highlighted sentences. When you revisit it again, summarize the bolded sentences in a few bullet points at the top. Each pass through the content distills it further, creating layers of compression. The original full text remains available for deep dives; the summaries provide fast access for recall.
This maps almost exactly to the problem of agent memory management. An agent's first pass through information produces detailed notes. A compression step produces a summary of those notes. If context pressure increases, the summary can be further compressed. The original can be archived to external storage for retrieval if needed.
Soar's chunking mechanism (Laird et al., 1987) was an early formal approach to procedural compression: compiling successful reasoning traces into compact production rules, effectively compressing the work of problem-solving into a quick-access form. The analogy to agent memory compression is direct.
MemGPT (Packer et al., 2023) brought this formally into the LLM agent context. MemGPT's architecture treats the context window as "main memory" and external storage as a "disk." It implements virtual memory: when main memory fills up, the agent can page content to disk and retrieve it later. The compression of main memory content before paging is explicit - the agent summarizes content before archiving it, ensuring that important signals are preserved even if the full detail is lost.
Generative Agents (Park et al., 2023) from Stanford demonstrated compression in a different form: agents that store observations in a stream and periodically run a "reflection" step that synthesizes high-level insights from recent observations. These reflections are then stored alongside raw observations. During retrieval, reflections score highly by both recency and importance, naturally rising to the top. This is hierarchical compression with emergent prioritization.
The Compression Hierarchy
Effective memory compression for agents operates across four levels. Understanding each level helps you design a system that preserves the right information at the right granularity.
Level 1 - Working Buffer: The current conversation turn and immediate context. Always in context, full fidelity. Typically 1,000-3,000 tokens. This is what the agent is actively thinking about right now.
Level 2 - Session Summary: A rolling summary of the current session. Updated periodically (every N turns or when a significant decision is made). Typically 3,000-8,000 tokens. This is the "recent memory" - what happened in the last hour or so. High detail on recent events, compressed but still rich.
Level 3 - Episode Archive: Summaries of earlier session segments that have been compressed to make room for newer content. Each archive entry might cover two to four hours of work in 500-1,000 tokens. These are retrieved and injected when relevant to the current task.
Level 4 - External Storage: The full raw content - documents read, tool outputs generated, earlier reasoning traces. Stored in a vector database, retrieved on demand when the agent needs to access specific content from the distant past.
Core Implementation - The Memory Manager
# memory_manager.py
import anthropic
import json
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
import chromadb
client = anthropic.Anthropic()
@dataclass
class MemoryEntry:
"""A single entry in the agent's memory stream."""
entry_id: str
content: str
entry_type: str # "observation", "reasoning", "decision", "tool_result"
importance: float # 0.0 to 1.0 - set at creation and updated by reflection
timestamp: str = field(default_factory=lambda: datetime.utcnow().isoformat())
compressed: bool = False
tags: list[str] = field(default_factory=list)
@property
def token_estimate(self) -> int:
"""Rough estimate: 4 characters per token."""
return len(self.content) // 4
@dataclass
class MemoryState:
"""Complete memory state for one agent session."""
session_id: str
task_description: str
# Level 1: Working buffer - recent entries at full fidelity
working_buffer: list[MemoryEntry] = field(default_factory=list)
# Level 2: Session summary - compressed but detailed
session_summary: str = ""
# Level 3: Episode archives - highly compressed earlier segments
episode_archives: list[dict] = field(default_factory=list)
# Configuration
working_buffer_token_limit: int = 4000 # Tokens before compression triggers
session_summary_token_limit: int = 8000 # Tokens before archival triggers
target_compression_ratio: float = 0.3 # Compress to ~30% of original
class MemoryManager:
"""
Manages agent memory across a long-running session.
Implements hierarchical compression to keep important context accessible.
"""
def __init__(self, state: MemoryState, vector_store_path: str = "./memory_db"):
self.state = state
self.chroma = chromadb.PersistentClient(path=vector_store_path)
self.archive_collection = self.chroma.get_or_create_collection(
name=f"session_{state.session_id}"
)
def add_entry(
self,
content: str,
entry_type: str = "observation",
importance: float = 0.5,
tags: list[str] | None = None
) -> MemoryEntry:
"""Add a new entry to working memory, triggering compression if needed."""
import uuid
entry = MemoryEntry(
entry_id=str(uuid.uuid4())[:8],
content=content,
entry_type=entry_type,
importance=importance,
tags=tags or []
)
self.state.working_buffer.append(entry)
# Check if working buffer needs compression
buffer_tokens = sum(e.token_estimate for e in self.state.working_buffer)
if buffer_tokens > self.state.working_buffer_token_limit:
self._compress_working_buffer()
# Check if session summary needs archival
summary_tokens = len(self.state.session_summary) // 4
if summary_tokens > self.state.session_summary_token_limit:
self._archive_session_summary()
return entry
def _compress_working_buffer(self):
"""
Compress older working buffer entries into the session summary.
Preserves high-importance entries at full fidelity; compresses the rest.
"""
print("[Memory] Compressing working buffer...")
# Sort buffer by importance and recency
# Keep the most recent 20% and all high-importance entries at full fidelity
buffer = self.state.working_buffer
keep_recent_count = max(5, len(buffer) // 5)
recent_entries = buffer[-keep_recent_count:]
older_entries = buffer[:-keep_recent_count]
# High-importance entries are kept even if old
high_importance = [e for e in older_entries if e.importance >= 0.8]
compress_entries = [e for e in older_entries if e.importance < 0.8]
if not compress_entries:
return # Nothing to compress
# LLM-based compression of older entries
compress_text = "\n\n".join([
f"[{e.entry_type.upper()} - importance {e.importance:.1f}]\n{e.content}"
for e in compress_entries
])
compression_prompt = f"""Compress the following agent memory entries into a concise summary.
CRITICAL REQUIREMENTS:
1. Preserve all important decisions and their reasoning
2. Keep all specific numbers, names, dates, and file paths
3. Preserve any constraints or requirements established
4. Compress routine observations that do not affect future decisions
5. Target: approximately {int(len(compress_text) * self.state.target_compression_ratio)} characters
Memory entries to compress:
{compress_text}
Write the compressed summary now:"""
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=2048,
messages=[{"role": "user", "content": compression_prompt}]
)
compressed_summary = response.content[0].text
# Update session summary by appending compressed content
if self.state.session_summary:
self.state.session_summary += f"\n\n---\n\n{compressed_summary}"
else:
self.state.session_summary = compressed_summary
# Archive the raw entries to vector store for potential retrieval
self._archive_to_vector_store(compress_entries)
# Rebuild working buffer: high-importance old + recent
self.state.working_buffer = high_importance + recent_entries
print(f"[Memory] Compressed {len(compress_entries)} entries "
f"({len(compress_text)} chars -> {len(compressed_summary)} chars, "
f"{len(compressed_summary)/len(compress_text):.0%} ratio)")
def _archive_session_summary(self):
"""
When the session summary grows too large, compress it further and archive.
The archive stores a high-level narrative; the vector store has the full detail.
"""
print("[Memory] Archiving session summary...")
archival_prompt = f"""This is a session summary from an ongoing agent task.
Compress it into a high-level archive entry that captures:
- Key decisions made and their rationale
- Important findings and facts discovered
- Current state of the task and what remains to do
- Critical constraints or requirements
Current session summary:
{self.state.session_summary}
Write a compressed archive entry (aim for ~25% of current length):"""
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=2048,
messages=[{"role": "user", "content": archival_prompt}]
)
archive_entry = {
"timestamp": datetime.utcnow().isoformat(),
"content": response.content[0].text,
"original_length": len(self.state.session_summary),
"compressed_length": len(response.content[0].text)
}
self.state.episode_archives.append(archive_entry)
# Store original summary in vector DB before clearing
self.archive_collection.add(
ids=[f"summary_{datetime.utcnow().timestamp()}"],
documents=[self.state.session_summary],
metadatas=[{"type": "session_summary", **archive_entry}]
)
# Reset session summary to just the archive entry for continuity
self.state.session_summary = (
f"[Archive Entry {len(self.state.episode_archives)}]\n"
f"{archive_entry['content']}"
)
print(f"[Memory] Archived summary "
f"({archive_entry['original_length']} -> "
f"{archive_entry['compressed_length']} chars)")
def _archive_to_vector_store(self, entries: list[MemoryEntry]):
"""Store raw entries in vector database for future retrieval."""
if not entries:
return
self.archive_collection.add(
ids=[e.entry_id for e in entries],
documents=[e.content for e in entries],
metadatas=[{
"entry_type": e.entry_type,
"importance": e.importance,
"timestamp": e.timestamp,
"tags": ",".join(e.tags)
} for e in entries]
)
def retrieve_relevant_archived_content(
self,
query: str,
n_results: int = 3
) -> list[str]:
"""Retrieve relevant content from the vector archive."""
try:
results = self.archive_collection.query(
query_texts=[query],
n_results=n_results
)
return results["documents"][0] if results["documents"] else []
except Exception:
return []
def build_context_for_agent(
self,
current_query: str = "",
max_tokens: int = 12000
) -> str:
"""
Build the memory context to inject into the agent's prompt.
Includes session summary, episode archives, and working buffer.
Retrieves relevant archived content if query is provided.
"""
sections = []
# Always include task description
sections.append(f"## Task\n{self.state.task_description}")
# Include episode archives (highly compressed older context)
if self.state.episode_archives:
archive_text = "\n\n".join([
f"**Episode {i+1}** ({a['timestamp'][:10]}):\n{a['content']}"
for i, a in enumerate(self.state.episode_archives)
])
sections.append(f"## Earlier Work (Archived)\n{archive_text}")
# Include session summary (compressed recent context)
if self.state.session_summary:
sections.append(f"## Recent Session Summary\n{self.state.session_summary}")
# Retrieve relevant archived content if query provided
if current_query:
relevant = self.retrieve_relevant_archived_content(current_query)
if relevant:
retrieval_text = "\n\n---\n\n".join(relevant[:2])
sections.append(
f"## Relevant Retrieved Context\n{retrieval_text}"
)
# Include working buffer (most recent, full fidelity)
if self.state.working_buffer:
buffer_text = "\n\n".join([
f"[{e.entry_type}] {e.content}"
for e in self.state.working_buffer[-20:] # Last 20 entries
])
sections.append(f"## Current Working Context\n{buffer_text}")
full_context = "\n\n".join(sections)
# Enforce token budget
token_estimate = len(full_context) // 4
if token_estimate > max_tokens:
# Truncate from the middle sections (preserve task + working buffer)
# This is a simple truncation - a more sophisticated implementation
# would compress the middle sections further
print(f"[Memory] Context too large ({token_estimate} tokens estimated), truncating...")
full_context = full_context[:max_tokens * 4]
return full_context
Progressive Summarization for Agents
Tiago Forte's progressive summarization applied to agent memory: each compression pass distills the most important information, creating layers you can navigate.
# progressive_summarizer.py
import anthropic
client = anthropic.Anthropic()
class ProgressiveSummarizer:
"""
Implements Forte's progressive summarization for agent memory.
Layer 1: Full content
Layer 2: Key sentences highlighted (reduced to ~40%)
Layer 3: Most important highlighted sentences bolded (reduced to ~15%)
Layer 4: Executive summary at top (50-100 words)
"""
def summarize_layer2(self, content: str) -> str:
"""Extract the most important sentences (layer 2 - ~40% of original)."""
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=2048,
messages=[{
"role": "user",
"content": f"""Extract the most important sentences from the following content.
Keep sentences that contain: decisions, specific numbers/data, constraints, findings,
errors encountered, and conclusions. Remove: filler transitions, restatements,
routine observations without specific data.
Target: retain approximately 40% of the original content.
Content:
{content}
Output only the extracted important sentences, maintaining their original wording:"""
}]
)
return response.content[0].text
def summarize_layer3(self, layer2_content: str) -> str:
"""From layer 2, extract the single most critical points (layer 3 - ~15%)."""
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=1024,
messages=[{
"role": "user",
"content": f"""From the following key sentences, identify the MOST critical points.
These are the points that are absolutely essential for understanding the current state
of the task and making future decisions. Target: approximately 15% of original length.
Key sentences:
{layer2_content}
Critical points only:"""
}]
)
return response.content[0].text
def create_executive_summary(self, layer3_content: str) -> str:
"""Create a 50-100 word executive summary from the critical points."""
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=256,
messages=[{
"role": "user",
"content": f"""Write a 50-100 word executive summary from these critical points.
The summary should answer: What is the current state? What key decisions were made?
What are the most important constraints or findings?
Critical points:
{layer3_content}
Executive summary (50-100 words):"""
}]
)
return response.content[0].text
def compress(self, content: str, target_layer: int = 3) -> dict[str, str]:
"""
Apply progressive summarization to content.
Returns all layers for selective retrieval.
"""
layers = {"layer1": content}
if target_layer >= 2:
layers["layer2"] = self.summarize_layer2(content)
if target_layer >= 3:
layers["layer3"] = self.summarize_layer3(layers["layer2"])
if target_layer >= 4:
layers["layer4"] = self.create_executive_summary(layers["layer3"])
return layers
# Demonstration
summarizer = ProgressiveSummarizer()
long_research_content = """
I began analyzing the target company's SEC 10-K filing for fiscal year 2023.
The company reported total revenue of $284.7 million, representing 23% year-over-year growth.
Gross margin improved from 61.2% to 67.4% driven primarily by increased software subscription revenue mix.
Operating expenses increased 34% to $198.3 million, resulting in an operating loss of $6.8 million.
The company has $142 million in cash and cash equivalents with a 28-month runway at current burn rate.
Customer count grew from 1,847 to 2,634 (42% growth). Net Revenue Retention was reported at 118%.
The company has three major enterprise customers each representing more than 10% of revenue -
this concentration risk is flagged explicitly in risk factors.
I noted that the company's largest customer, referred to as Customer A, is understood to be Megacorp Inc
based on cross-referencing with their public vendor announcements.
The company is currently in discussions with two potential strategic partners in the European market.
Legal proceedings section notes a patent dispute filed in March 2023 by CompetitorX regarding
their core data processing algorithm - this is a material risk not reflected in current valuation.
"""
result = summarizer.compress(long_research_content, target_layer=4)
print("Layer 1 (Full):", len(result["layer1"]), "chars")
print("Layer 2 (Key sentences):", len(result["layer2"]), "chars",
f"({len(result['layer2'])/len(result['layer1']):.0%})")
print("Layer 3 (Critical points):", len(result["layer3"]), "chars",
f"({len(result['layer3'])/len(result['layer1']):.0%})")
print("Layer 4 (Executive summary):", len(result["layer4"]), "chars",
f"({len(result['layer4'])/len(result['layer1']):.0%})")
print("\nExecutive Summary:\n", result["layer4"])
Lossy vs. Lossless Compression
Not all information compresses equally. Understanding the tradeoff is essential for designing what to preserve and what to discard.
The key insight: specific data is precious, connective tissue is cheap. A compressed memory that says "revenue $284.7M, gross margin 67.4%, NRR 118%, 3 customers >10% revenue concentration, patent dispute with CompetitorX" is infinitely more useful than one that says "the company showed strong growth with improving margins and some customer concentration risks."
Long-Running Agent with Automatic Compression
A complete agent that runs for multiple hours without losing critical context:
# long_running_agent.py
import anthropic
import json
import uuid
from memory_manager import MemoryManager, MemoryState, MemoryEntry
client = anthropic.Anthropic()
def run_long_running_agent(
task: str,
max_iterations: int = 100,
verbose: bool = True
) -> dict:
"""
Agent that can run for hours without context overflow.
Memory is actively managed with compression and archival.
"""
session_id = str(uuid.uuid4())[:8]
state = MemoryState(
session_id=session_id,
task_description=task,
working_buffer_token_limit=3000,
session_summary_token_limit=6000,
target_compression_ratio=0.3
)
memory = MemoryManager(state, vector_store_path=f"./memory_{session_id}")
# Log initial task
memory.add_entry(
content=f"Task started: {task}",
entry_type="observation",
importance=1.0
)
tools = [
{
"name": "read_document",
"description": "Read a document or web page",
"input_schema": {
"type": "object",
"properties": {
"source": {"type": "string", "description": "URL or file path"}
},
"required": ["source"]
}
},
{
"name": "write_finding",
"description": "Record an important finding (high importance - always preserved)",
"input_schema": {
"type": "object",
"properties": {
"finding": {"type": "string"},
"importance": {
"type": "number",
"description": "0.0 to 1.0 - how critical is this finding?"
}
},
"required": ["finding", "importance"]
}
},
{
"name": "search_my_memory",
"description": "Search through archived memory for relevant past findings",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "What to search for"}
},
"required": ["query"]
}
},
{
"name": "complete_task",
"description": "Signal that the task is complete and provide the final result",
"input_schema": {
"type": "object",
"properties": {
"result": {"type": "string", "description": "Final task output"}
},
"required": ["result"]
}
}
]
messages = []
final_result = None
total_turns = 0
# Main agent loop
for iteration in range(max_iterations):
total_turns += 1
# Build memory context for this turn
# Use the last message as the query for relevant archived content retrieval
current_query = messages[-1]["content"][-1]["content"] if messages else task
if isinstance(current_query, list):
# Handle tool result format
current_query = str(current_query[0].get("content", task))[:200]
memory_context = memory.build_context_for_agent(
current_query=current_query,
max_tokens=10000
)
system_prompt = f"""You are an expert research and analysis agent.
You have access to memory management tools to preserve important findings.
MEMORY STATE:
{memory_context}
INSTRUCTIONS:
- Use write_finding for any important fact, number, decision, or constraint
(importance 0.8-1.0 for critical, 0.4-0.7 for useful, 0.1-0.3 for routine)
- Use search_my_memory if you need context from earlier in the session
- Work systematically - the task may take many steps
- Call complete_task only when you have fully finished the task"""
if not messages:
messages = [{"role": "user", "content": task}]
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=4096,
system=system_prompt,
tools=tools,
messages=messages
)
if response.stop_reason == "end_turn":
for block in response.content:
if hasattr(block, "text"):
final_result = block.text
memory.add_entry(
content=f"Task completed: {block.text[:500]}",
entry_type="decision",
importance=1.0
)
break
if response.stop_reason == "tool_use":
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type != "tool_use":
continue
tool_name = block.name
tool_input = block.input
result_content = ""
if tool_name == "write_finding":
finding = tool_input["finding"]
importance = float(tool_input.get("importance", 0.5))
memory.add_entry(
content=finding,
entry_type="decision",
importance=importance
)
result_content = f"Finding recorded (importance {importance:.1f})"
if verbose:
print(f"[Finding {importance:.1f}] {finding[:100]}")
elif tool_name == "search_my_memory":
query = tool_input["query"]
archived = memory.retrieve_relevant_archived_content(query, n_results=3)
if archived:
result_content = "Relevant archived memories:\n\n" + "\n\n---\n\n".join(archived)
memory.add_entry(
content=f"Memory retrieval: {query}",
entry_type="observation",
importance=0.2
)
else:
result_content = "No relevant archived memories found."
elif tool_name == "read_document":
source = tool_input["source"]
# In real implementation: actually fetch the document
simulated_content = f"[Content from {source}: simulated for demo]"
result_content = simulated_content
memory.add_entry(
content=f"Read document: {source}",
entry_type="observation",
importance=0.3
)
elif tool_name == "complete_task":
final_result = tool_input["result"]
result_content = "Task marked as complete."
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result_content
})
messages.append({"role": "user", "content": tool_results})
if final_result:
break
if verbose and iteration % 10 == 0 and iteration > 0:
# Log memory statistics every 10 turns
buffer_tokens = sum(e.token_estimate for e in state.working_buffer)
summary_tokens = len(state.session_summary) // 4
print(f"\n[Memory Stats at turn {iteration}]")
print(f" Working buffer: {len(state.working_buffer)} entries, ~{buffer_tokens} tokens")
print(f" Session summary: ~{summary_tokens} tokens")
print(f" Episode archives: {len(state.episode_archives)}")
print(f" Archived entries in vector store: {state.session_id}\n")
return {
"session_id": session_id,
"total_turns": total_turns,
"final_result": final_result,
"memory_stats": {
"working_buffer_entries": len(state.working_buffer),
"episode_archives": len(state.episode_archives),
"session_summary_length": len(state.session_summary)
}
}
if __name__ == "__main__":
result = run_long_running_agent(
task=(
"Perform comprehensive due diligence on Acme Corp, a B2B SaaS company "
"considering a $50M Series B investment. Analyze: financial metrics, "
"market position, competitive landscape, team quality, and key risks. "
"Produce a 3-page investment memo."
),
max_iterations=50,
verbose=True
)
print("\n=== Final Result ===")
print(result["final_result"])
print("\n=== Memory Stats ===")
print(json.dumps(result["memory_stats"], indent=2))
What to Always Preserve vs. Compress vs. Evict
This decision framework drives the importance scoring that powers the compression hierarchy.
# importance_scorer.py
import anthropic
import json
client = anthropic.Anthropic()
IMPORTANCE_SYSTEM = """You are evaluating agent memory entries for importance.
Score from 0.0 to 1.0 based on these criteria:
HIGH IMPORTANCE (0.8-1.0):
- Specific numbers, measurements, or metrics
- Key decisions and their explicit rationale
- Established constraints or requirements that must be respected
- Errors or failures that affected the task trajectory
- Names of key people, systems, files, or organizations central to the task
- Critical findings that change the understanding of the task
MEDIUM IMPORTANCE (0.4-0.7):
- General observations relevant to the task
- Context that may be needed later
- Intermediate results that could be re-derived but are useful to have
LOW IMPORTANCE (0.1-0.3):
- Process steps that were routine and expected
- Confirmations that things worked as expected
- Transitional or navigational steps
NEAR ZERO (0.0-0.1):
- Filler language and transitions
- Restatements of already-known information
- Exploratory steps that led nowhere
Respond with JSON: {"score": 0.0-1.0, "reason": "brief explanation"}"""
def score_memory_importance(content: str, task_context: str = "") -> float:
"""Score the importance of a memory entry for compression decisions."""
prompt = f"Task context: {task_context[:200]}\n\nMemory entry to score:\n{content}"
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=128,
system=IMPORTANCE_SYSTEM,
messages=[{"role": "user", "content": prompt}]
)
try:
result = json.loads(response.content[0].text)
return float(result.get("score", 0.5))
except (json.JSONDecodeError, ValueError):
return 0.5 # Default to medium importance if scoring fails
MemGPT's Virtual Memory System
MemGPT treats the context window as RAM and external storage as disk, implementing virtual memory paging for agents.
The genius of MemGPT is exposing memory management operations as agent tools. The agent actively decides when to page content to disk and when to retrieve it. This is better than passive compression because the agent has task context that the compression system lacks - it knows which archived content will be relevant for the next step.
Production Engineering Notes
Compression costs real money. Every compression call is an LLM API call with input tokens equal to the content being compressed. Budget for this explicitly. In practice, a compression call happens every 20-30 agent turns for most tasks. At 3,000-5,000 input tokens per compression call, this adds 10-20% overhead to total token costs. The tradeoff is worth it for long-running tasks; for short tasks (under 30 turns), compression may add more overhead than it saves.
Never compress within 10 turns of starting. Early in a task, all context is potentially important and the agent does not have enough history to know what is routine vs. critical. Set a minimum turn count before compression triggers.
Deduplicate before compressing. Working buffers frequently contain the same fact mentioned multiple times across different turns. A deduplication pass before compression significantly improves compression quality. Simple substring matching or embedding similarity can identify near-duplicates.
Save state to disk after every compression event. Compression is expensive; do not redo it if the agent crashes. Serialize the full memory state (working buffer, session summary, episode archives) to disk after every compression. This also enables warm restarts - the topic of the next lesson.
Test your compression quality. After each compression, ask the model: "Based only on this compressed summary, what were the three most important findings from the original content?" Compare the answer to the ground truth. If critical information is being lost, adjust your compression prompt or lower the compression ratio.
:::danger Compression Destroys Information by Design
Compression is lossy. Accept this. The goal is not to preserve everything - it is to preserve everything important. The danger arises when the importance scoring is wrong: if routine observations are scored as high-importance, the compression system will not compress them and context will fill up anyway. If critical decisions are scored as low-importance, they will be lost.
Tune your importance scoring carefully. Test it with real task traces. Have humans review compression outputs during initial deployment to catch systematic importance scoring errors before they cause production failures. :::
:::warning The "Lost in the Middle" Problem Persists Even with Compression
Even with good compression, the context window has an attention distribution problem: the model attends more strongly to content at the beginning and end of the context than to content in the middle. For long contexts, place the most critical memory - the session summary and high-importance facts - at the beginning of the system prompt, and the most recent working buffer at the end. Content that is less critical can go in the middle where attention is weakest.
Also be aware that compression-generated summaries can introduce hallucinations. The compression model may infer information that was not in the original. Always prompt compression calls with explicit instructions against fabrication: "Only include information explicitly stated in the content. Do not infer or extrapolate." :::
Interview Q&A
Q1: What is the "lost in the middle" problem and why does it matter for agent memory design?
Liu et al. (2023) demonstrated through systematic experimentation that language models perform significantly worse at retrieving information placed in the middle of a long context compared to information at the beginning or end. In a 128K context window, information at position 64K (the middle) may be effectively inaccessible to the model even though it is technically within the context window. This has direct implications for agent memory design. Naive approaches that concatenate all history in chronological order place the most important context - the task briefing and early key findings - at the beginning, where it will initially be highly attended. But as more content is added, important middle-section content loses attention weight. Memory managers should actively manage placement: keep the most critical context (task description, key constraints, latest findings) at the extremes of the context - beginning and end - and place routine intermediate content in the middle where attention is weakest.
Q2: How does progressive summarization differ from simple truncation or sliding window approaches?
Simple truncation removes the oldest content entirely - it is fast but discards information without regard for its importance. A sliding window maintains the most recent N tokens but has the same problem: old-but-critical context (the task briefing from the start of the session) is discarded as newer content arrives. Progressive summarization, derived from Tiago Forte's personal knowledge management approach, creates a hierarchy of compressions: full content → key sentences (40%) → critical points (15%) → executive summary (5%). Each compression layer is preserved alongside the more compressed versions. An agent can access the executive summary for fast orientation, the critical points for key facts, or the full content for detailed reference. No information is destroyed - it is moved to an archived tier (external vector storage) while compressed representations remain in-context. The key difference is selectivity: progressive summarization preserves high-importance content at higher fidelity while aggressively compressing low-importance content.
Q3: Describe MemGPT's virtual memory architecture and its advantages over passive memory management.
MemGPT treats the LLM context window as RAM and external storage as disk, implementing virtual memory paging explicitly. The agent has four memory tiers: system prompt (procedures and persona), core memory (always-in-context key facts, editable by the agent), recall storage (recent conversation, automatically paged to disk as it grows), and archival memory (searchable vector database). Crucially, MemGPT exposes memory operations as agent tools: archival_memory_insert() to page content to disk, archival_memory_search() to retrieve relevant content, and core_memory_replace() to update key facts. The advantage over passive memory management is that the agent has task context that the memory system lacks. When the agent has just learned a critical fact that it knows will be needed 50 turns later, it can explicitly store it in archival memory with a descriptive tag - not wait for a passive compression system to decide whether to preserve it. This gives the agent agency over its own memory, which tends to produce better preservation of task-critical information.
Q4: What is the right compression ratio for agent memory, and how do you choose it?
There is no universal right answer - compression ratio is a function of the information density of the content being compressed. Research notes and analytical reasoning are typically compressible to 20-30% of original length while preserving essential information. Conversation and tool-use logs that contain much routine content (confirmations, navigation steps) may compress to 10-15%. Raw document text being summarized by the agent may compress to 5-10%. The practical approach is to start with a target of 25-30% and measure compression quality empirically: after each compression, generate questions from the original content, then answer them using only the compressed version, and measure the accuracy. If critical information is being lost (accuracy drops significantly), reduce the compression ratio. If the compressed summaries are much longer than they need to be, increase it. Different content types should have different target ratios - implementing a content-type-aware compression ratio is more sophisticated but produces significantly better results.
Q5: How do you handle the tradeoff between compression cost (LLM API calls) and context window overflow?
The key insight is that compression is not free - every compression call consumes LLM tokens and adds latency. For short tasks (under 20 turns), compression overhead can exceed the benefits. The practical approach: implement compression as a triggered operation with a minimum turn count (never compress before turn 15) and a minimum content size (never compress less than 2,000 tokens). Use a cheaper, faster model for compression - Claude Haiku or a similar model handles summarization well at much lower cost than Claude Opus. Cache compression prompts where possible using Anthropic's prompt caching (the system instructions for compression do not change between calls). Finally, implement predictive compression: rather than waiting for the context window to fill, trigger compression when utilization exceeds 60% of the target limit. This gives the system time to compress gracefully before reaching crisis conditions, and the resulting summaries are higher quality because the model has headroom to think carefully rather than being forced to rush.
Q6: What should always be preserved in agent memory, regardless of compression pressure?
Four categories of content should survive all compression passes at high fidelity: (1) Specific numbers and measurements - dollar amounts, percentages, counts, dates, version numbers. "Revenue $284.7M" compresses to "strong revenue" in poor compression; the specific number was the point. (2) Decisions and their explicit rationale - not just "we chose approach X" but "we chose approach X because of constraint Y and rejected approach Z due to risk W." Rationale is often discarded in poor compression but is critical for understanding future decisions. (3) Established constraints and requirements - "must use API v2," "cannot modify the production database," "the customer requires weekly reports." Constraints established early in a task must survive to the end; violating them due to compression loss is a serious failure mode. (4) Named entities central to the task - specific person names, company names, file paths, system names. Vague references ("the main file," "the external API") are often substituted during poor compression and cause ambiguity. These four categories should be explicitly tagged as high-importance (0.9-1.0) at creation time so the compression system is biased to preserve them.