hao-ai-lab/LookaheadDecoding

DFlash: Block Diffusion for Flash Speculative Decoding

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

FlashInfer: Kernel Library for LLM Serving

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

Entropy Based Sampling and Parallel CoT Decoding

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

A high-throughput and memory-efficient inference and serving engine for LLMs

TokenSpeed is a speed-of-light LLM inference engine.