1CatAI/1Cat-vLLM-Gaudi

Community maintained hardware plugin for vLLM on Intel Gaudi

V100 / SM70-focused vLLM engineering fork for modern LLM inference.

Easy and lightning fast training of 🤗 Transformers on Habana Gaudi processor (HPU)

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

A high-throughput and memory-efficient inference and serving engine for LLMs - Optimized for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60

~1300 line, minimal, extensible LLM inference engine built from scratch.

A high-throughput and memory-efficient inference and serving engine for LLMs

The vLLM XPU kernels for Intel GPU

A highly optimized LLM inference acceleration engine for Llama and its variants.