thu-ml/SageAttention

Fast and memory-efficient exact attention

FlashMLA: Efficient Multi-head Latent Attention Kernels

🚀 Efficient implementations for emerging model architectures

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Transformer related optimization, including BERT, GPT

FlashInfer: Kernel Library for LLM Serving

Hackable and optimized Transformers building blocks, supporting a composable construction.

Fast inference engine for Transformer models

DFlash: Block Diffusion for Flash Speculative Decoding