predibase/lorax

A high-throughput and memory-efficient inference and serving engine for LLMs

Cost-efficient and pluggable Infrastructure components for GenAI inference

Achieve state of the art inference performance with modern accelerators on Kubernetes

Universal LLM Deployment Engine with ML Compilation

A Datacenter Scale Distributed Inference Serving Framework

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

High-speed Large Language Model Serving for Local Deployment

Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.