InferMatrixLab/vllm-rlt

A high-throughput and memory-efficient inference and serving engine for LLMs

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

~1300 line, minimal, extensible LLM inference engine built from scratch.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Efficient AI Inference & Serving

Based on Nano-vLLM, a simple replication of vLLM with self-contained paged attention and flash attention implementation

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

Pure Rust Inference Engine