scaleapi/llm-engine

Achieve state of the art inference performance with modern accelerators on Kubernetes

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

A high-throughput and memory-efficient inference and serving engine for LLMs

Universal LLM Deployment Engine with ML Compilation

Cost-efficient and pluggable Infrastructure components for GenAI inference

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading