EricLBuehler/candle-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

High-speed Large Language Model Serving for Local Deployment

A fast inference library for running LLMs locally on modern consumer-class GPUs

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

A simple TUI for serving local LLM models. Pick a model, pick a backend, serve it

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

llama.cpp fork with additional SOTA quants and improved performance

Control panel for VLLM, Sglang, llama.cpp, exllamav3