HKUSTDial/flash-sparse-attention

🚀🚀 Efficient implementations of Native Sparse Attention

🚀 Efficient implementations for emerging model architectures

Fast and memory-efficient exact attention

MoBA: Mixture of Block Attention for Long-Context LLMs

A collection of memory efficient attention operators implemented in the Triton language.

Helpful tools and examples for working with flex-attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

FlashMLA: Efficient Multi-head Latent Attention Kernels

Hackable and optimized Transformers building blocks, supporting a composable construction.