kvmem/kvmem

LLM inference in C/C++

Give any local LLM a billion-token memory. Open-source, local-first context + memory engine for Ollama and AI agents — runs on your own machine, free. 🧠⚡

OrbitKV is a KV cache for vLLM and SGLang.

Smart OpenAI‑compatible proxy for llama.cpp: manages slots, saves/restores KV cache to disk, routes requests by prefix similarity, and protects hot slots from being overwritten. Accelerates long prompts (30–60k tokens) via instant reuse or fast on‑demand restore; supports SSE streaming and non‑stream JSON over /v1/chat/completions.

Run the full 2.78-trillion-parameter Kimi K3 model, DeepSeek V4.1 Flash or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

Galahad by Corbenic

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

LLM inference in C/C++