Emerging-AI/ENOVA

Enterprise-grade LLM automated deployment tool that makes AI servers truly "plug-and-play".

Universal LLM Deployment Engine with ML Compilation

Scale LLM Engine public repository

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

A high-throughput and memory-efficient inference and serving engine for LLMs

Achieve state of the art inference performance with modern accelerators on Kubernetes

Selfhost modern LLM stacks. Run the whole fleet from your terminal

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Cost-efficient and pluggable Infrastructure components for GenAI inference