Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Abstract

Standard transformer attention uses identical dimensionality for queries, keys, and values, yet these components serve different roles: queries and keys produce scalar attention weights (selection), while values carry rich representations (value transfer). We show that selection requires only O( N) dimensions to distinguish among N relevant token categories (e.g., syntactic roles, semantic clusters, positional patterns) -- far fewer than value transfer needs. We introduce factored keys, which exploit this asymmetry to physically shrink the KV cache of any pretrained model without retraining from scratch -- unlike GQA and MLA, which must be designed into the architecture before pretraining. We factorize each key projection W_K A_d r B_r d via truncated SVD (where r = d_select), set W_K' = A as the new key projection producing compact r-dimensional keys for the cache, and absorb B^ into the query projection (W_Q' = W_Q B^) at zero cost -- since queries are never cached. At 7B scale, training from scratch with r = d_model/4 matches full-attention perplexity (9.2 vs 9.3 PPL after 20B tokens) while using 12% fewer parameters and training 8% faster. For existing models, SVD + QK fine-tuning (3 epochs, less than 1% of pretraining data) achieves 75% key cache savings at approximately 2% quality cost on both GPT-2 and Mistral-7B. The approach composes with GQA and quantization for up to 16 combined key cache compression. For a 7B model serving 128K context, factored keys save 25 GB of KV cache per user, enabling approximately 60% more concurrent users on identical hardware.

Reproductions