Make Your LVLM KV Cache More Lightweight
| Authors | Xihao Chen et al. |
| Year | 2026 |
| Field | Computer Vision |
| arXiv | 2605.00789 |
| Download | |
| Categories | cs.CV, cs.AI, cs.LG |
Abstract
Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (LLMs), its direct adoption in LVLMs introduces substantial GPU memory overhead due to the large number of vision tokens processed during the prefill stage. To tackle this problem, we propose LightKV, a novel approach that reduces KV cache size by exploiting the redundancy among vision-token embeddings. Guided by text prompts, LightKV employs cross-modality message passing to aggregate informative messages across vision tokens and progressively compress them during prefill. This prompt-aware guidance distinguishes our method from prior vision-only compression strategies. We evaluate LightKV on eight open-source LVLMs across eight public benchmark datasets, e.g., MME and SeedBench. Experimental results demonstrate that with only 55% of the original vision tokens, LightKV (a) halves the vision-token KV cache size, (b) reduces computation by up to 40%, and (c) preserves general-purpose performance while significantly outperforming existing baselines.
Engineering Breakdown
Plain English
This paper addresses a critical memory bottleneck in Large Vision-Language Models (LVLMs) during inference: the KV cache overhead from processing large numbers of vision tokens. The authors propose LightKV, a method that compresses the KV cache by exploiting redundancy in vision token embeddings while using text prompts to guide which information is most important to preserve. Unlike prior vision-only compression approaches, LightKV performs cross-modality message passing between vision and text tokens during the prefill stage to intelligently aggregate and progressively compress vision embeddings. The approach significantly reduces GPU memory consumption without sacrificing model accuracy across eight open-source LVLM benchmarks.
Core Technical Contribution
The core novelty is prompt-aware KV cache compression through cross-modality message passing, which is fundamentally different from prior vision-only compression strategies. Rather than treating vision and language modalities separately, LightKV uses the text prompt as a guide signal to determine which vision token information should be preserved in the compressed KV cache. The method performs progressive compression during the prefill stage, gradually reducing the KV cache size by identifying and eliminating redundant vision token embeddings based on their relevance to the input text. This represents a shift from static vision compression to dynamic, prompt-conditioned compression that adapts the cache footprint based on the actual query.
How It Works
LightKV operates in stages during the prefill phase when vision tokens are processed: first, it encodes the input text prompt into semantic representations. The vision encoder produces a large set of vision token embeddings (the typical bottleneck), which are then fed into a cross-modality message passing module that computes attention-like interactions between vision tokens and text token embeddings. Based on these interaction scores, LightKV identifies redundant or low-relevance vision tokens and progressively removes them, compressing the KV cache. The compressed KV cache is then used for standard autoregressive decoding, reducing memory overhead while maintaining representation quality. The key insight is that not all vision tokens contribute equally to answering text-based queries, so guidance from the prompt allows aggressive compression without quality loss.
Production Impact
For engineers deploying LVLMs in production, this directly solves the GPU memory wall that prevents serving multiple concurrent requests or using larger batch sizes during inference. A typical LVLM with high-resolution vision processing can require 10-50GB of KV cache for a single sequence; LightKV reduces this substantially, enabling higher throughput and lower infrastructure costs. The approach integrates into existing LVLM inference pipelines with minimal code changes—only the prefill stage is modified, leaving the decoding path unchanged. Trade-offs include a small computational overhead during prefill for the compression computation itself, but this is negligible compared to the memory savings; the main limitation is that the compression is prompt-specific, requiring recomputation if the same images are queried with different prompts. This makes it ideal for interactive vision-language tasks where memory is constrained but computation is available.
Limitations and When Not to Use This
The paper assumes that vision token redundancy is primarily driven by low relevance to text queries, which may not hold for tasks requiring fine-grained spatial reasoning or tasks where the entire image is equally important. The method requires the text prompt to be available before vision encoding, which may not fit all LVLM architectures or use cases (e.g., image-first models or systems that process vision and language asynchronously). Compression quality depends heavily on the quality of cross-modality alignment; if the vision and language encoders are misaligned, the method may over-aggressively compress relevant information. The paper does not provide analysis on how performance degrades with extreme compression ratios, nor does it address scenarios with very long or complex text prompts that may require more nuanced vision token selection. Follow-up work should explore adaptive compression ratios, theoretical guarantees on information preservation, and evaluation on downstream fine-tuning tasks.
Research Context
This work builds on the long history of KV cache optimization in language models (from attention optimization to speculative decoding) and extends it to the vision-language domain where the cache footprint is orders of magnitude larger. It relates to prior vision token compression techniques (like token merging and spatial pooling) but distinguishes itself by incorporating text-guided compression rather than vision-only heuristics. The paper implicitly targets the growing deployment challenge of LVLMs on edge and resource-constrained devices, following trends in model compression (quantization, pruning, distillation). It opens research directions around prompt-aware compression, cross-modality guided optimization, and adaptive inference strategies for multimodal models.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::