MDK8888/GPTFast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Accessible large language models via k-bit quantization for PyTorch.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

A collection of tricks and tools to speed up transformer models

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

A fast inference library for running LLMs locally on modern consumer-class GPUs