llm-d/llm-d-router

Achieve state of the art inference performance with modern accelerators on Kubernetes

Gateway API Inference Extension

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

A high-performance and light-weight router for vLLM large scale deployment

vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

Cost-efficient and pluggable Infrastructure components for GenAI inference

High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.

The open source control plane for AI inference