inferflow/inferflow

A high-throughput and memory-efficient inference and serving engine for LLMs

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Efficient AI Inference & Serving

Universal LLM Deployment Engine with ML Compilation

High-speed Large Language Model Serving for Local Deployment

Scale LLM Engine public repository

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Large-scale LLM inference engine