The risk of KV cache compression

arXiv (Cornell University)

Abstract

Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache. The prevalent approach to this bottleneck is KV cache compression, which replaces the full cache with a compact summary. Despite its practical importance, the design of such summaries is largely driven by empirical experimentation. On the theoretical side, existing results show that KV cache compression can be impossible in the worst case, but offer little systematic guidance for designing algorithms in regimes where accurate compression is possible. We bridge this gap by characterizing the minimax risk of KV cache compression in terms of the intrinsic compressibility of a cache, revealing when and how accurate compression is possible. These results yield novel design principles for KV cache compression under causal masking that map efficiently to prefill and autoregressive decoding while achieving minimax-optimal risk. We instantiate these principles in a practical algorithm and report promising performance on LongBench in targeted experiments. Overall, our results provide a principled avenue for practical KV cache compression with theoretical guarantees.

Authors 5

1. Lukas Haverbeck Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

2. Carmen Amo Alonso
3. Andrés Felipe Posada-Moreno Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

4. Sebastian Trimpe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

5. Marco Pavone

   Stanford University

   Affiliation as printed

   Stanford University