SafeAILab/EAGLE

DFlash: Block Diffusion for Flash Speculative Decoding

DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

A fast inference library for running LLMs locally on modern consumer-class GPUs

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

A high-throughput and memory-efficient inference and serving engine for LLMs