vllm-project/compressed-tensors

A Lossless Compression Library for AI pipelines

State-of-the-art LLM compression, built for production inference with vLLM

Python package for LLM compression

Simple, safe way to store and distribute tensors

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

For releasing code related to compression methods for transformers, accompanying our publications

LLM KV cache compression made easy

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

Official Pytorch repository for Extreme Compression of Large Language Models via Additive Quantization https://arxiv.org/pdf/2401.06118.pdf and PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression https://arxiv.org/abs/2405.14852