meta-pytorch/gpt-fast

A fast inference library for running LLMs locally on modern consumer-class GPUs

Accessible large language models via k-bit quantization for PyTorch.

A high-throughput and memory-efficient inference and serving engine for LLMs

High-speed Large Language Model Serving for Local Deployment

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

Python bindings for llama.cpp

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Fast, flexible LLM inference

The official PyTorch implementation of Google's Gemma models