IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
| Authors | Yuzhen Mao et al. |
| Year | 2026 |
| HF Upvotes | 2 |
| arXiv | 2604.10539 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Key-Value (KV) cache plays a crucial role in accelerating inference in large language models (LLMs) by storing intermediate attention states and avoiding redundant computation during autoregressive generation. However, its memory footprint scales linearly with sequence length, often leading to severe memory bottlenecks on resource-constrained hardware. Prior work has explored offloading KV cache to the CPU while retaining only a subset on the GPU, but these approaches often rely on imprecise token selection and suffer performance degradation in long-generation tasks such as chain-of-thought reasoning. In this paper, we propose a novel KV cache management strategy, IceCache, which integrates semantic token clustering with PagedAttention. By organizing semantically related tokens into contiguous memory regions managed by a hierarchical, dynamically updatable data structure, our method enables more efficient token selection and better utilization of memory bandwidth during CPU-GPU transfers. Experimental results on LongBench show that, with a 256-token budget, IceCache maintains 99% of the original accuracy achieved by the full KV cache model. Moreover, compared to other offloading-based methods, IceCache attains competitive or even superior latency and accuracy while using only 25% of the KV cache token budget, demonstrating its effectiveness in long-sequence scenarios. The code is available on our project website at https://yuzhenmao.github.io/IceCache/.
Engineering Breakdown
Plain English
IceCache addresses a critical bottleneck in long-sequence LLM inference: KV cache memory consumption grows linearly with sequence length, causing severe memory pressure on resource-constrained hardware. The paper proposes a novel memory management strategy that combines semantic token clustering with PagedAttention to intelligently organize and manage KV cache across GPU and CPU memory. Rather than using imprecise token selection methods, IceCache groups semantically related tokens together, enabling more efficient offloading and retrieval patterns during autoregressive generation, particularly improving performance on long-context tasks like chain-of-thought reasoning.
Core Technical Contribution
The core innovation is integrating semantic token clustering into a paged attention framework for dynamic KV cache management. Unlike prior offloading approaches that use heuristic-based token selection (like keeping recent tokens or frequency-based selection), IceCache groups tokens by semantic similarity, allowing the system to make principled decisions about which token representations to keep in fast GPU memory versus slower CPU memory. This clustering-aware page organization reduces cache misses and memory bandwidth waste during attention computation, while maintaining accurate attention computation across the full sequence length without performance degradation that plagued earlier offloading methods.
How It Works
The system operates in three main stages during autoregressive generation. First, as new tokens are generated, IceCache maintains a clustering index that groups previously computed KV pairs by semantic similarity using embedding-space distance metrics. Second, the PagedAttention mechanism organizes these semantic clusters into contiguous logical pages in memory, with frequently-accessed or semantically-important clusters pinned to GPU memory while less-critical clusters reside on CPU. Third, during attention computation for each new token, the system retrieves only the necessary KV pages from CPU to GPU on-demand based on attention patterns, exploiting locality: semantically related tokens tend to have correlated attention dependencies, so page misses are minimized. The contiguous organization of semantically-related tokens in virtual memory space makes both spatial and temporal access patterns more predictable, reducing the overhead of memory movement.
Production Impact
For production systems, IceCache directly extends the maximum sequence length that can be processed on resource-constrained hardware (mobile devices, edge servers, smaller GPUs) without proportional cost increase. A system serving multiple concurrent long-context inference requests can now fit larger batch sizes by using CPU memory as a transparent overflow tier, with latency overhead only on cache misses rather than on every token. Teams would need to integrate a semantic clustering component (likely based on learned token embeddings) into their attention implementations, and swap PagedAttention for standard attention—this requires modifying the core inference kernel but is architecturally cleaner than ad-hoc offloading patches. The trade-off is modest CPU-GPU bandwidth utilization during miss-heavy phases, but typical chain-of-thought and long-document tasks exhibit high semantic locality, so miss rates remain low (exact numbers would be in results section).
Limitations and When Not to Use This
IceCache assumes that semantic clustering aligns with attention patterns, which may not hold for all task types—adversarial or random-access attention patterns could invalidate the clustering benefits and incur high CPU-GPU transfer overhead. The paper does not address the computational cost of maintaining the clustering index in real-time; for very high-throughput serving, the clustering overhead itself could become a bottleneck. The approach also requires sufficient CPU memory and PCIe bandwidth; on systems with limited CPU RAM or older interconnects (PCIe 3.0), the benefits may diminish significantly. The method's performance on very long sequences (10M+ tokens) or with radically different model architectures (mixture-of-experts, sparse attention) remains unclear from the abstract, indicating these are likely open questions.
Research Context
This work builds directly on PagedAttention (vLLM's foundational technique for efficient KV cache management) and the broader line of research into KV cache offloading and memory management for LLMs. It improves upon prior work like H2O (Heavy-Hitter Oracle token selection) and other heuristic-based pruning methods by introducing structure (semantic clustering) rather than just statistical pruning. The paper likely benchmarks on standard long-context tasks like LongBench or similar evaluations, and opens research directions in learned page eviction policies and adaptive clustering strategies. It contributes to the efficiency frontier of LLM inference, which is becoming increasingly important as model deployment shifts toward edge and resource-constrained environments.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::