STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

AIPR assessment

The problem is hard and highly competitive, with many groups optimizing KV-cache memory, compression, and latency in parallel. The paper's strengths compound well: adaptive rank learning, hybrid key/value decomposition, quantization, and fused kernels all attack the same bottleneck, and the empirical story is consistent across quality, long-context, and throughput metrics. The main weaknesses also interact: the method needs calibration and custom kernel support, and some speed claims depend on e

Abstract

Low-rank projection has emerged as a promising approach for compressing the KV cache by exploiting hidden-dimension redundancy. However, prior methods rely on fixed or heuristic rank selection and struggle to achieve aggressive compression with minimal accuracy degradation. We propose STAR-KV, an adaptive low-rank KV cache compression framework with fine-grained rank control. STAR-KV encompasses 1) a differentiable thresholding mechanism that enables optimal rank selection at both attention-head and block levels, 2) a hybrid decomposition strategy that applies different low-rank factorizations according to the sensitivity of key and value projections, and 3) a low-rank-aware mixed precision quantization that leverages data statistics for near lossless low-bit quantization. Evaluated across multiple LLMs and benchmarks, STAR-KV achieves up to 75% KV cache compression and up to 20x overall KV cache reduction when combined with quantization. Enabled by custom Triton-based GPU kernels, STAR-KV delivers up to 6.9x speedup for the attention module and 3.1x end-to-end generation throughput. Our code is publicly available at: https://github.com/PriyanshBhatnagar/STAR-KV.

Score Breakdown

More from this week

• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents
• GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines
• vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models
• ScaleDisturb: Exploiting Temporal Asymmetry to Amplify Read Disturbance in Modern DRAM Chips
• The CIFAR Synthetic Evidence Corpus for Detecting AI-Generated Evidence

More in Machine Learning

• Verifiable and Confidential DNN Inference on Low-End Edge Devices
• HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
• LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems
• TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
• DynMuon: A Dynamic Spectral Shaping View of Muon