Cross-Modal Variance-Aware KV Cache Optimization for Efficient Multimodal Long-Context Inference

Electronics, vol. 15, pp. 3206

Abstract

Multimodal large language models (MLLMs) face substantial memory bottlenecks when processing long visual contexts, such as videos and high-resolution images. Existing methods that allocate visual KV cache budgets using cross-modal attention entropy mainly estimate the distributional breadth of text–vision interaction and may overlook how visual relevance varies across query positions in the encoded multimodal context. We propose a cross-modal query-position variance-aware KV cache optimization method for efficient multimodal long-context inference. The proposed method combines cross-modal attention entropy with prefill-stage query-position variance computed from cross-modal attention to estimate layer-wise visual KV cache preferences. Based on this preference score, visual KV cache budgets are allocated across layers, and variance-aware token pruning is applied to retain high-importance KV states while directly evicting redundant visual tokens without feature merging. Experiments on the MileBench benchmark using LLaVA-v1.5-7B show that, under deterministic single-run evaluation and while retaining only 20% of the visual KV cache, the proposed method produces point-estimate performance close to the full-cache reference and higher point estimates on several fine-grained reasoning and retrieval subtasks. Additional representative-subtask evaluations under different visual cache budgets and on InternVL2.5-8B further provide preliminary point-estimate evidence that the proposed allocation signal is not restricted to a single cache ratio or backbone. System profiling further shows that the 20% cache setting reduces measured KV cache GPU memory from 1.28 GiB to 0.26 GiB and decoding latency from 100.28 ms/token to 92.85 ms/token. These results suggest that cross-modal query-position variance may help preserve sparse, query-dependent visual cues under low-cache-budget multimodal inference.

Authors 3

1. S Liu

   State Grid Corporation of China (China)

   Affiliation as printed

   Network and Data Security Department, State Grid Information & Telecommunication Center (Big Data Center), Beijing 100761, China

2. S AN corresponding

   State Grid Corporation of China (China)

   Affiliation as printed

   Network and Data Security Department, State Grid Information & Telecommunication Center (Big Data Center), Beijing 100761, China

3. Yiwei Ru

   Chinese Academy of Sciences · Institute of Automation

   Affiliation as printed

   Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China