BlinkDL/Albatross

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Pure Rust Inference Engine

Inference RWKV with multiple supported backends.

One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing and Volta NVIDIA GPUs, consolidated from the forks that carry it: GGUF block formats, ternary Bonsai 2, device route profiles, MTP and DFlash2.

Qwen3.8-27B on one RTX 4090: full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots

Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds.

From-scratch C++/CUDA inference engine for Qwen3-8B

Based on Nano-vLLM, a simple replication of vLLM with self-contained paged attention and flash attention implementation