local-inference-lab/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

Efficient AI Inference & Serving

The official API server for Exllama. OAI compatible, lightweight, and fast.

Large-scale LLM inference engine

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

Scale LLM Engine public repository

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

High-speed Large Language Model Serving for Local Deployment