tile-ai/TileRT

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

A high-throughput and memory-efficient inference and serving engine for LLMs

High-speed Large Language Model Serving for Local Deployment

Run the full 2.78-trillion-parameter Kimi K3 model, DeepSeek V4.1 Flash or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

TokenSpeed is a speed-of-light LLM inference engine.

Runs 405B LLMs on 8GB VRAM

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle