openlake-project/openlake

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

FlashInfer: Kernel Library for LLM Serving

Train and serve LLMs at extreme speed and massive throughput.

~1300 line, minimal, extensible LLM inference engine built from scratch.

AIStore: scalable storage for AI applications

Model Express is a Rust-based component meant to be placed next to existing model inference systems to speed up their startup times and improve overall performance.

Pure Rust Inference Engine

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

High Performance KV Cache for LLM Inference