neutree-ai/neutree

Cost-efficient and pluggable Infrastructure components for GenAI inference

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

Enterprise-grade LLM automated deployment tool that makes AI servers truly "plug-and-play".

Scale LLM Engine public repository

Achieve state of the art inference performance with modern accelerators on Kubernetes

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

✈️ Kubernetes-native platform for deploying and managing AI inference across multiple providers

🏗️ Fine-tune, build, and deploy open-source LLMs easily!