VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

AIPR assessment

This is a hard, competitive systems problem, not an uncrowded niche: KV-cache reduction, speculative decoding, and long-context serving have all been actively optimized by many groups. The paper's strengths reinforce each other well, a clean mechanism, a convincing resource argument, and measured throughput gains on real hardware. The weaknesses also compound, since the lack of a public artifact, limited ablation, and a few citation gaps make the strongest claims harder to independently verify.

Abstract

The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy-despite minimal accuracy degradation for short outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded, which leads to catastrophic failures in code generation and tool calling. We present VeriCache, the first inference framework that ensures the same output as full-KV-cache decoding but largely preserves the high decoding throughput of a range of KV cache compression algorithms. VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. While it may seem like just speculative decoding, VeriCache requires addressing a key system challenge to work-keeping the full KV cache out of GPU memory and minimizing the overhead of swapping it in for verification. The insight is two-fold: (1) compressed-KV decoding can be parallelized with full-KV swap, because one is HBM-bandwidth-bound and the other is PCIe/network-bound, and (2) the compressed KV cache often produces output similar to the full KV cache, allowing a long drafting horizon to amortize each full-KV swap. VeriCache applies to both long-context decoding and remote prefix caching, supports a broad family of token-dropping and quantization methods through a uniform compressor interface, and composes with traditional speculative decoding. Experimental results show that VeriCache achieves up to 4X higher throughput than full-KV inference while producing identical outputs.

Score Breakdown

More from this week

• TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
• ContraFix: Agentic Vulnerability Repair via Differential Runtime Evidence and Skill Reuse
• Loaded Dice: Solving the Non-Selection Problem for Scalable Probabilistic RowHammer Defense
• Stop Starving or Stuffing Me: Boosting Firmware Fuzzing Efficiency with On-demand Input Delivery
• DynMuon: A Dynamic Spectral Shaping View of Muon

More in Computer Architecture

• ScaleDisturb: Exploiting Temporal Asymmetry to Amplify Read Disturbance in Modern DRAM Chips
• ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
• Optimus: Elastic Decoding for Efficient Diffusion LLM Serving
• DORA: Dataflow-Instruction Orchestration Architecture for DNN Acceleration
• Loaded Dice: Solving the Non-Selection Problem for Scalable Probabilistic RowHammer Defense