microsoft/LLMLingua

State-of-the-art LLM compression, built for production inference with vLLM

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

Optimizing inference proxy for LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.