intel/intel-extension-for-transformers

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

🤗 Optimum Intel: Accelerate inference with Intel optimization tools

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Universal LLM Deployment Engine with ML Compilation

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

High-speed Large Language Model Serving for Local Deployment

A high-throughput and memory-efficient inference and serving engine for LLMs