ai-dynamo/kvcr

KV cache store for distributed LLM inference

Distributed KV cache scheduling & offloading libraries

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

High Performance KV Cache for LLM Inference

Persist and reuse KV Cache to speedup your LLM.

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

Object-storage-native KV cache for LLM inference & RL. Cross-restart, cross-conversation, cross-engine via shared S3 bucket.