InternLM/lmdeploy

Universal LLM Deployment Engine with ML Compilation

A high-throughput and memory-efficient inference and serving engine for LLMs

High-speed Large Language Model Serving for Local Deployment

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

LLM inference in C/C++

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.