PiKV: KV Cache Management System for Mixture of Experts

Code

• github.com/noakliu/pikv
  OfficialIn paper★ 49
• github.com/noakliu/pikvpress
  OfficialIn paper★ 6

Abstract

As large language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce PiKV, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages expert-sharded KV storage to partition caches across GPUs, PiKV routing to reduce token-to-KV access, and a PiKV Scheduling to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates PiKV Compression modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: https://github.com/NoakLiu/PiKV. Experiments details is recorded at: https://github.com/NoakLiu/PiKV/Experimental\_Results. We also have PiKV integrated with Nvidia kvpress for acceleration, details see https://github.com/NoakLiu/PiKVpress. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.

Reproductions