InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models
| Authors | Sayed Mohammadreza Tayaranian Hosseini et al. |
| Year | 2026 |
| Field | Machine Learning |
| arXiv | 2602.23200 |
| Download | |
| Categories | cs.LG, cs.CL |
Abstract
Reducing the hardware footprint of large language models (LLMs) during decoding is critical for efficient long-sequence generation. A key bottleneck is the key-value (KV) cache, whose size scales with sequence length and easily dominates the memory footprint of the model. Previous work proposed quantization methods that are focused on compressing the KV cache while maintaining its information. We introduce InnerQ, a hardware-aware KV-cache quantization scheme that lowers decode latency without sacrificing accuracy. InnerQ applies group-wise quantization while grouping the cache matrices over their inner dimension. Unlike previous work that group over the outer dimension, InnerQ aligns dequantization with the vector-matrix multiplication and enables scale factor reuse across GPU compute units. This reduces memory accesses and accelerates dequantization, yielding up to speedup over previous work and up to over half-precision vector-matrix multiplication. To preserve fidelity under aggressive compression, InnerQ incorporates (i) hybrid quantization, selecting symmetric or asymmetric quantization per group based on local statistics; (ii) high-precision windows for both the most recent tokens and the attention sink tokens to mitigate outlier leakage; and (iii) per-channel normalization of the key cache, computed once during prefill and folded into the query to avoid runtime overhead. Our evaluation experiments on Llama models shows that InnerQ maintains a few-shot GSM8K performance comparable to non-quantized KV caches and surpasses prior KV cache quantization methods.
Engineering Breakdown
Plain English
InnerQ solves a critical bottleneck in deploying large language models: the key-value (KV) cache during decoding consumes massive memory that scales with sequence length, slowing inference on long contexts. The paper proposes a hardware-aware quantization scheme that compresses the KV cache by grouping quantization along the inner dimension of cache matrices rather than the outer dimension used in prior work. This alignment with how GPUs perform vector-matrix multiplication enables scale factor reuse across compute units and reduces decode latency without accuracy loss. The approach is tuning-free, meaning it requires no training or calibration—it applies quantization strategies that work efficiently across different hardware.
Core Technical Contribution
InnerQ's core novelty is changing the quantization grouping dimension from outer (sequence-length dimension) to inner (feature/embedding dimension), which fundamentally changes how the dequantized cache interacts with GPU matrix multiplication hardware. By grouping along the inner dimension, the scale factors can be reused across multiple GPU warps or thread blocks performing the same vector-matrix multiply, reducing memory bandwidth and enabling hardware-level optimizations that prior outer-dimension quantization schemes could not exploit. The insight is that quantization strategy should be co-designed with the actual compute pattern—specifically, the attention computation where (batch × sequence) queries multiply against quantized (sequence × head-dimension) keys and values. This is a hardware-aware approach rather than a pure algorithmic one, treating quantization as an optimization problem that must account for GPU execution patterns.
How It Works
InnerQ operates on the KV cache matrices which have shape (sequence_length, num_heads × head_dim) for keys and values in typical transformer implementations. During quantization, instead of grouping rows (outer dimension, representing different tokens), InnerQ groups along columns (inner dimension, representing the embedding feature space). For each group of features, a single scale factor is computed and applied during dequantization. When attention computation happens, the dequantized KV cache interacts with query projections through matrix multiplication: attention_scores = Query @ Key^T and attention_output = attention_scores @ Value. Because the inner-dimension grouping aligns with how GPU cores are organized (threads compute across the feature dimension), the scale factors needed for dequantization are already in the right cache lines and registers, enabling reuse across multiple queries or batches. The quantization is applied post-training with no retraining or calibration needed, making deployment straightforward.
Production Impact
For engineers deploying LLMs in production, InnerQ directly reduces memory consumption of KV caches, enabling either longer sequence lengths on fixed GPU memory or smaller, cheaper GPU SKUs to handle the same workload. In a typical production setup with batch-size=32 and sequence-length=4096, the KV cache can consume 30-40% of GPU memory for large models (70B+ parameters); quantizing this from FP32 or FP16 to INT8 or lower immediately frees significant capacity. Decode latency improves because less data moves across the GPU memory hierarchy—the scale factors are reused locally rather than fetched repeatedly, reducing memory bandwidth pressure during attention computation. Integration is minimal: the quantization can be applied as a preprocessing step on cached KV tensors without modifying the attention kernel itself, though vendor-specific kernels (CUTLASS, TensorRT) could further optimize dequantization. The trade-off is accuracy: quantization introduces rounding error, and the paper must demonstrate that inner-dimension grouping maintains model outputs within acceptable tolerances (typically <0.5% top-1 accuracy drop).
Limitations and When Not to Use This
The paper's abstract indicates the approach is hardware-aware, but it's unclear which hardware it optimizes for—the techniques likely need retuning for different GPU architectures (NVIDIA H100 vs A100 vs AMD MI300), and the benefits may not transfer to other accelerators like TPUs or AWS Trainium chips. InnerQ assumes a standard transformer attention pattern; it may not generalize to sparse attention, multi-query attention, or other attention variants that change the compute-to-memory ratio. The tuning-free claim suggests fixed quantization bit-widths and grouping strategies, but production systems often need adaptive quantization based on layer importance (early layers may tolerate more quantization than later ones)—it's unclear if InnerQ supports this. Additionally, the approach quantizes KV cache but likely not queries or weights, leaving optimization opportunities on the table and potentially creating bottlenecks in the interaction between unquantized and quantized tensors.
Research Context
This work builds on a decade of quantization research starting from QAT (Quantization-Aware Training) and post-training quantization methods, then progressing to activation-only quantization schemes like ZipML and GPTQ that avoid retraining. Prior KV cache quantization efforts (like KIVI or related works) quantized along the outer dimension but missed the hardware efficiency insight that inner-dimension grouping enables. The paper contributes to the emerging field of hardware-software co-design for LLM inference, where quantization strategies are evaluated not just on mathematical criteria (information preservation) but on actual GPU execution time and memory bandwidth. This work likely motivates follow-up research on hardware-aware quantization for other attention variants (FlashAttention patterns, multi-query attention), mixed-precision quantization across layers, and automatic selection of grouping strategies per GPU architecture.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::