LLMServe/DistServe

A high-throughput and memory-efficient inference and serving engine for LLMs

A Datacenter Scale Distributed Inference Serving Framework

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

Efficient AI Inference & Serving

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

Pipeline-parallel LLM inference across GPUs on separate machines.

Achieve state of the art inference performance with modern accelerators on Kubernetes

FlashInfer: Kernel Library for LLM Serving