vllm-project/router

llm-d Router: The intelligent entry point for inference requests

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

An open, programmable decision layer for models and compute.

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

The minimal interpretable model router that learns & adapts to your agent workflows

Self-hosted LLM router with a managed safety net. OpenAI-compatible. BYOK. Single-workspace. Streaming. For more advanced routing choose hosted OrcaRouter

Unified large model proxy gateway, providing load balancing, fault transfer, current limiting, budget management, content auditing, caching, and MCP gateway capabilities.

Simple CLI tool for deterministic routing of queries between local and hosted LLM models

High-performance Go proxy gateway for 9Router LLM routing — 32K RPS, 42MB RAM, CGO-free