turboderp-org/exllamav2

High-speed Large Language Model Serving for Local Deployment

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

Python bindings for llama.cpp

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

A high-throughput and memory-efficient inference and serving engine for LLMs

FlashInfer: Kernel Library for LLM Serving

Open source inference engine for agents that optimizes itself for your exact hardware. Compiles and tunes its kernels on your device, so open models run up to 2x faster than llama.cpp. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.