Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
| Authors | Zhenting Wang et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2603.04257 |
| Download | |
| Categories | cs.CL, cs.LG |
Abstract
Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.
Engineering Breakdown
Plain English
This paper addresses a fundamental bottleneck in LLM agents: their inability to effectively use long task trajectories because context windows fill up quickly with tool outputs and intermediate reasoning steps. The authors introduce Memex, an indexed experience memory system that maintains a compact working context with structured summaries and stable indices, while keeping full evidence accessible through efficient retrieval—essentially creating a hybrid working memory plus retrieval system instead of discarding past information. The key innovation is that Memex compresses context without losing evidence, unlike existing truncation or summarization approaches that permanently discard information. This allows agents to handle significantly longer-horizon tasks while maintaining access to distant evidence when needed.
Core Technical Contribution
The core contribution is the indexed experience memory architecture that decouples context compression from evidence loss. Rather than truncating trajectories or creating lossy summaries (which throw away details), Memex maintains two parallel structures: (1) a compact working context containing concise summaries and semantic indices, and (2) a full indexed evidence store that can be retrieved on demand. The mechanism uses structured summarization to create queryable representations of past experiences, paired with efficient retrieval (likely semantic or sparse indexing) to surface relevant evidence when the agent needs it. This is fundamentally different from prior approaches because it treats memory management as a retrieval problem rather than a compression problem.
How It Works
The system operates with a dual-memory design: as an LLM agent executes tasks and accumulates tool outputs and reasoning steps, Memex continuously creates structured summaries (condensed, semantic representations of what happened) and stores the full trajectory in an indexed evidence store. The working context window stays manageable by including only the summary and stable indices rather than raw evidence. When the agent needs information from past steps—to reason about what it tried, what failed, or what succeeded—it uses the indices to retrieve specific evidence from the full store and temporarily load it into context. The retrieval mechanism likely uses semantic similarity or structured queries (e.g., "find tool outputs from step 15") to efficiently locate relevant past experiences without exhaustively searching the entire trajectory. This process repeats throughout the task, with the indexed store growing while the working context remains bounded.
Production Impact
For teams building LLM agents in production, Memex solves the critical problem of handling multi-step tasks that would normally exhaust context budgets—enabling agents to work on tasks with dozens or hundreds of steps without degrading reasoning quality. In a production pipeline, you'd implement two storage layers: a vector or semantic index for summaries (e.g., using a dense retriever like DPR or an embedding database like Pinecone), and a full evidence store (could be in-memory, Redis, or a database). The latency trade-off is important: retrieving evidence adds per-query overhead (probably 50-500ms depending on index size), but this is offset by the ability to use smaller context windows, which reduces per-token inference latency in the main LLM. This approach requires careful engineering around retrieval accuracy—if the index returns irrelevant evidence or misses critical information, agent performance degrades. Integration complexity is moderate: you need to instrument your agent to emit structured summaries and manage the evidence store lifecycle, but the core LLM inference loop changes minimally.
Limitations and When Not to Use This
The paper assumes that structured summarization can capture task-relevant information without losing critical details, which may not hold for complex reasoning where subtle intermediate steps matter—some tasks might genuinely require seeing the full trajectory. Retrieval-based systems are fundamentally vulnerable to retrieval failure: if the indexed memory returns incomplete or irrelevant results, the agent can't recover that lost context, potentially degrading performance below even truncation baselines. The approach likely requires careful tuning of the summarization strategy and index structure per task domain; a general-purpose summary that works for code tasks may fail for open-ended reasoning tasks. The paper doesn't clearly address scaling behavior: as trajectories grow to thousands of steps, maintaining efficient indices and retrieval becomes challenging, and the cost of indexing updates could offset efficiency gains.
Research Context
This work builds on a growing line of research addressing the context window bottleneck in LLM agents, extending prior work on in-context learning limitations and memory-augmented language models. It relates to existing techniques like retrieval-augmented generation (RAG) but applies them within a single task trajectory rather than external knowledge bases, and differs from summarization-based approaches (which lose information) by treating memory as a retrieval problem. The paper likely benchmarks on long-horizon task datasets like WebShop, ALFWorld, or custom RL environments where context window exhaustion is a real blocker. This opens research directions around optimal summarization strategies, hybrid retrieval mechanisms (combining dense and sparse methods), and theoretical understanding of when and why indexed memory outperforms simpler baselines.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::