volcano-sh/kthena

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Achieve state of the art inference performance with modern accelerators on Kubernetes

Cost-efficient and pluggable Infrastructure components for GenAI inference

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

Kubernetes AI Toolchain Operator

vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Scale LLM Engine public repository