triton-inference-server/tensorrtllm_backend

Efficient AI Inference & Serving

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

A high-throughput and memory-efficient inference and serving engine for LLMs

~1300 line, minimal, extensible LLM inference engine built from scratch.

FlashInfer: Kernel Library for LLM Serving

Tile-Based Runtime for Ultra-Low-Latency LLM Inference

Run the full 2.78-trillion-parameter Kimi K3 model, DeepSeek V4.1 Flash or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

Pure Rust Inference Engine

Disaggregated serving system for Large Language Models (LLMs).