KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
| Authors | Chuangtao Chen et al. |
| Year | 2026 |
| HF Upvotes | 8 |
| arXiv | 2604.13226 |
| Download | |
| HF Page | View on Hugging Face |
Abstract
Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-dependent: reusing a cached document in a new context requires recomputing KV states to account for shifts in attention distribution. Existing solutions such as CacheBlend, EPIC, and SAM-KV mitigate this issue by selectively recomputing a subset of tokens; however, they still incur non-negligible computational overhead (FLOPs) and increased Time-to-First-Token (TTFT) latency. In this paper, we propose KV Packet, a recomputation-free cache reuse framework that treats cached documents as immutable ``packets'' wrapped in light-weight trainable soft-token adapters, which are trained via self-supervised distillation to bridge context discontinuities. Experiments on Llama-3.1 and Qwen2.5 demonstrate that the proposed KV Packet method achieves near-zero FLOPs and lower TTFT than recomputation-based baselines, while retaining F1 scores comparable to those of the full recomputation baseline.
Engineering Breakdown
Plain English
KV Packet addresses a major inefficiency in LLM inference: standard Key-Value caches cannot be reused across different contexts without expensive recomputation of attention states. The paper proposes KV Packet, which treats cached documents as immutable "packets" and wraps them with lightweight trainable soft-token adapters that learn to handle context shifts through self-supervised distillation. This eliminates the need for recomputation entirely, reducing computational overhead and Time-to-First-Token latency compared to prior methods like CacheBlend, EPIC, and SAM-KV that still require selective token recomputation. The approach achieves significant speedups by making KV cache reuse truly context-independent.
Core Technical Contribution
The core innovation is reframing KV cache reuse as a soft-token adapter problem rather than a recomputation-mitigation problem. Instead of trying to selectively recompute which tokens need attention updates, KV Packet introduces learnable adapter tokens that sit between the frozen cached KV states and the new context, allowing the model to seamlessly bridge attention distribution shifts without touching the cached computation. This is fundamentally different from prior work (CacheBlend, EPIC, SAM-KV) which still perform partial recomputation—KV Packet achieves zero recomputation by treating the cache as an immutable artifact that adapters can interpret contextually. The adapters are trained via self-supervised distillation, meaning they learn to produce the same outputs as full recomputation would, but without actually recomputing.
How It Works
The mechanism works in three stages. First, when a document is cached, its full KV states are computed and frozen as an immutable "packet." Second, lightweight trainable soft-token adapters are inserted that can attend to both the frozen KV packet and the new query context, allowing them to reweight and reinterpret the cached KV states for the new attention distribution. Third, during inference on a new context, instead of recomputing KV states, the adapter tokens run a forward pass to translate the cached KV packet into the appropriate representation for the new context's attention pattern. The adapters are trained via self-supervised distillation: they learn to mimic the outputs that would come from a full KV recomputation, using the recomputed KV states as the target. The key architectural insight is that adapters can learn context-dependent transformations of fixed cache embeddings, eliminating the need to touch the cached computation itself.
Production Impact
For production LLM serving systems, KV Packet directly reduces latency and compute costs for document/context reuse scenarios, which are increasingly common in RAG pipelines, multi-turn dialogue, and batch inference where the same source documents appear across multiple queries. The elimination of recomputation means fewer FLOPs per inference step, lower memory bandwidth requirements, and critically, lower Time-to-First-Token (TTFT)—a key metric in user-facing LLM applications. Integration is relatively straightforward: you cache documents once with their full KV states, store the adapter parameters separately (which are lightweight compared to full KV caches), and apply the adapter in the forward pass without modifying existing KV cache infrastructure. Trade-offs include the overhead of training adapters for each cached document type and the memory cost of storing adapter weights, though these are likely offset by the amortized savings across multiple context uses. The approach is particularly valuable in high-throughput serving scenarios where the same documents are queried multiple times with different prompts or contexts.
Limitations and When Not to Use This
KV Packet assumes that context shifts can be adequately modeled by soft-token adapters, which may fail for extreme distribution shifts (e.g., querying the same document in vastly different semantic contexts) where the adapter cannot learn a generalizable transformation. The paper does not address how to handle very long documents or documents that were cached under different model configurations, which could require retraining or invalidation of adapters. There is an open question about adapter generalization: how well do adapters trained on one set of context types transfer to new, unseen context patterns—this could limit the practical reuse window before adapters need retraining. Additionally, the self-supervised distillation training process requires access to ground-truth recomputed KV states, which adds a one-time offline cost; the paper does not fully characterize the compute required for this training phase or optimal strategies for when to retrain adapters.
Research Context
This work builds directly on a lineage of KV cache optimization research (CacheBlend, EPIC, SAM-KV) that identified recomputation as a bottleneck in context-dependent cache reuse, but diverges by proposing adapters rather than selective recomputation. It contributes to the broader efficiency research direction in LLM inference, alongside quantization and pruning, by attacking a different angle: making cache reuse truly zero-recomputation. The soft-token adapter mechanism is inspired by recent work in parameter-efficient fine-tuning (LoRA, prompt tuning) but applied to a novel problem domain—inference-time cache adaptation. This opens new research directions around learned cache transformations, adapter generalization across contexts, and the theoretical foundations of why soft tokens can effectively bridge KV cache contexts.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::