microsoft/vidur

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Open Source Inference Research Platform Standard / 开源推理研究平台

Everything you need to know about LLM inference

A high-throughput and memory-efficient inference and serving engine for LLMs

~1300 line, minimal, extensible LLM inference engine built from scratch.

Offline optimization of your disaggregated Dynamo graph

100 days of LLM inference engineering — daily posts, experiments, and visualizations

TokenSpeed is a speed-of-light LLM inference engine.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.