turboderp-org/exllamav3

A fast inference library for running LLMs locally on modern consumer-class GPUs

High-speed Large Language Model Serving for Local Deployment

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

llama.cpp fork with additional SOTA quants and improved performance

LLM speculative inference server for heterogeneous hardware & consumer GPUs

Open source inference engine for agents that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.

LLM inference in C/C++

Python bindings for llama.cpp

A high-throughput and memory-efficient inference and serving engine for LLMs