xLLM-AI/xllm

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

AiTer Optimized Model

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

A high-throughput and memory-efficient inference and serving engine for LLMs

Pure Rust Inference Engine

A highly optimized LLM inference acceleration engine for Llama and its variants.