bitsandbytes-foundation/bitsandbytes

AirLLM 70B inference with single 4GB GPU

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

DeepGEMM: clean and efficient BLAS kernel library on GPU

Official inference framework for 1-bit LLMs

High-speed Large Language Model Serving for Local Deployment

A high-throughput and memory-efficient inference and serving engine for LLMs