Tiiny-AI/PowerInfer

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

A fast inference library for running LLMs locally on modern consumer-class GPUs

Run PyTorch LLMs locally on servers, desktop and mobile

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the hardware you already own.

Zig INferenCe Engine — Local LLM inference on AMD GPUs and Apple Silicon

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

On-device LLM Inference Powered by X-Bit Quantization

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware