TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

AIPR assessment

Problem difficulty is high. KV-cache compression for long-context, agentic inference is a crowded and competitive area, and the paper targets realistic workloads where memory and throughput are both constrained. The strengths reinforce each other: the taxonomy is simple, the gains are measured on real serving stacks, and the throughput improvements line up with the memory savings. The weaknesses also compound: the method depends on offline calibration and template-specific tagging, there is no p

Abstract

Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bitwidths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\times$ KV cache size and achieving 30% higher end-to-end throughput, when running on real GPU systems.

Score Breakdown

More from this week

• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
• ContraFix: Agentic Vulnerability Repair via Differential Runtime Evidence and Skill Reuse
• Loaded Dice: Solving the Non-Selection Problem for Scalable Probabilistic RowHammer Defense
• Stop Starving or Stuffing Me: Boosting Firmware Fuzzing Efficiency with On-demand Input Delivery
• DynMuon: A Dynamic Spectral Shaping View of Muon

More in Machine Learning

• STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
• Verifiable and Confidential DNN Inference on Low-End Edge Devices
• HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
• LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems
• DynMuon: A Dynamic Spectral Shaping View of Muon