abetlen/llama-cpp-python

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

High-speed Large Language Model Serving for Local Deployment

Fast, flexible LLM inference

A high-throughput and memory-efficient inference and serving engine for LLMs

LLM inference in C/C++

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

High-performance In-browser LLM Inference Engine

Universal LLM Deployment Engine with ML Compilation

Distribute and run LLMs with a single file.