rwilliamspbg-ops/Ghostlink

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

[Official] Prima.cpp: Scale Your Local AI Beyond One Device.

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

QuantumFlow - Distributed LLM inference scheduling framework with multi-backend support (vLLM, TGI, SGLang), adaptive scheduling strategies, and cluster management.

A Datacenter Scale Distributed Inference Serving Framework

Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.

Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.

An Efficient and Versatile Inference Engine for Distributed LLM Serving