InfiniTensor/InfiniLM

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

A high-throughput and memory-efficient inference and serving engine for LLMs

Pure Rust Inference Engine

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Efficient AI Inference & Serving

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.