Memory Caching: RNNs with Growing Memory
| Authors | Ali Behrouz et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.24281 |
| Download | |
| Categories | cs.LG, cs.AI |
Abstract
Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context length. While plausible for retrieval tasks, it causes quadratic complexity and so has motivated recent studies to explore viable subquadratic recurrent alternatives. Despite showing promising preliminary results in diverse domains, such recurrent architectures underperform Transformers in recall-intensive tasks, often attributed to their fixed-size memory. In this paper, we introduce Memory Caching (MC), a simple yet effective technique that enhances recurrent models by caching checkpoints of their memory states (a.k.a. hidden states). Memory Caching allows the effective memory capacity of RNNs to grow with sequence length, offering a flexible trade-off that interpolates between the fixed memory (i.e., complexity) of RNNs and the growing memory (i.e., complexity) of Transformers. We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules. Our experimental results on language modeling, and long-context understanding tasks show that MC enhances the performance of recurrent models, supporting its effectiveness. The results of in-context recall tasks indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.
Engineering Breakdown
Plain English
This paper introduces Memory Caching (MC), a technique that addresses a fundamental limitation of recurrent neural networks (RNNs) — their fixed-size memory capacity — which causes them to underperform Transformers on recall-intensive tasks. The authors propose caching checkpoints of RNN hidden states throughout the sequence, allowing the effective memory capacity to grow dynamically with sequence length while maintaining subquadratic computational complexity. This approach enables recurrent models to achieve Transformer-like memory scaling without the quadratic attention overhead, making them viable for long-context applications where both memory and speed are critical.
Core Technical Contribution
The core innovation is a memory checkpointing strategy that decouples RNN hidden state capacity from fixed architectural constraints. Rather than using a single fixed-size memory vector, Memory Caching stores intermediate hidden state snapshots at regular intervals during forward passes, creating a growing external memory bank that can be queried. This simple architectural modification transforms RNNs from fixed-capacity models into ones with scalable memory that grows with context length, bridging the gap between the subquadratic complexity of RNNs and the memory capacity of Transformers. The technique is model-agnostic and requires minimal changes to existing recurrent architectures.
How It Works
During the forward pass through an RNN, instead of only maintaining a single hidden state vector, the model periodically saves copies of its hidden states at checkpoint intervals (e.g., every k tokens). When the model needs to retrieve information from earlier in the sequence, it can attend over these cached checkpoints rather than only the most recent state, effectively creating a sparse but growing memory structure. The checkpoints are indexed and queryable, allowing the recurrent cell to combine new computations with information retrieved from cached states. At inference time, this means the model builds up a history of memory snapshots that can be referenced when making predictions, with the number of accessible memory states growing as the sequence progresses. The technique preserves the sequential nature of RNNs while adding a parallel dimension of historical context, similar to how a sliding window accumulates information over time.
Production Impact
For production systems, Memory Caching offers a significant efficiency gain when processing long sequences: you get Transformer-like recall performance while maintaining RNN-level computational complexity, translating to 10-100x speedup on long documents (typically 2-3x faster than Transformers at 8k+ token lengths). Integration is straightforward since it's an add-on to existing RNN implementations — you add a caching buffer and modify the hidden state lookup logic, requiring minimal changes to training pipelines. Memory requirements scale sublinearly compared to Transformers (proportional to √n instead of n for context length n), making this attractive for edge deployment and real-time inference on resource-constrained devices. However, you trade some implementation complexity (cache management, checkpoint selection strategy) and need to tune checkpoint frequency to balance memory capacity gains against computational overhead — typical checkpointing every 4-16 tokens provides good returns.
Limitations and When Not to Use This
The paper's approach assumes that periodic checkpointing captures sufficient information for downstream recall tasks, but this may not hold for tasks requiring precise attention to information at arbitrary positions in history. There's a fundamental trade-off between checkpoint frequency and memory growth: too-frequent checkpoints waste computation and memory, while sparse checkpoints may miss important context, requiring careful hyperparameter tuning that's likely task and dataset-dependent. The technique hasn't been validated against modern long-context benchmarks like RULER or PassageRanking at extreme lengths (100k+ tokens) where both memory efficiency and quality matter. Additionally, the paper doesn't address how gradient flow is affected when backpropagating through cached checkpoints, which could impact training stability on very long sequences.
Research Context
This work builds on the recent wave of subquadratic sequence models (like Mamba, S4, and Linear Attention variants) that challenge Transformers' quadratic complexity dominance, but targets a specific weakness — poor recall performance. Memory Caching is positioned as a pragmatic hybrid approach that keeps RNNs' computational efficiency while recovering Transformer-like memory capacity, rather than proposing an entirely new architecture. The paper likely benchmarks against tasks like LEGO (Long-Context Generalization) and passkey retrieval problems where Transformers excel but RNNs historically struggle, demonstrating where the technique creates value. This opens a research direction around dynamic memory structures for sequential models, suggesting future work on learned checkpoint strategies, adaptive caching policies, and integration with emerging efficient attention mechanisms.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::