sgl-project/sglang-jax

TPU inference for vLLM, with unified JAX and PyTorch support.

Accelerate, Optimize performance with streamlined training and serving options with JAX.

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

A high-throughput and memory-efficient inference and serving engine for LLMs

A simple, performant, and scalable Jax LLM!

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Universal LLM Deployment Engine with ML Compilation

The Triton TensorRT-LLM Backend

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.