GuglielmoCerri/khazad

Cut LLM costs by up to 80% and unlock sub-millisecond responses with intelligent semantic caching.A drop-in, provider-agnostic LLM proxy written in Go with sub-millisecond response

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

A reverse proxy server to cache the slow expensive requests to the openai api. Built with nginx and docker.

Unified multi-layer caching library for AI/agent pipelines — LangChain, LangGraph, AutoGen, CrewAI, Agno, A2A

Smart OpenAI‑compatible proxy for llama.cpp: manages slots, saves/restores KV cache to disk, routes requests by prefix similarity, and protects hot slots from being overwritten. Accelerates long prompts (30–60k tokens) via instant reuse or fast on‑demand restore; supports SSE streaming and non‑stream JSON over /v1/chat/completions.

Drop-in OpenAI-compatible cache that replays repeated computer-use and agent workflows at near-zero cost.

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Radicalbit AI Gateway

Production LLM call layer for AI agents and tools: keep OpenAI/Anthropic/AI SDK/LiteLLM, hot-swap models with MDA presets, and add cache, retries, circuit breakers, key rotation, singleflight, and Python/TypeScript/Rust parity.