ModelTC/LightLLM

A high-throughput and memory-efficient inference and serving engine for LLMs

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

TokenSpeed is a speed-of-light LLM inference engine.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.