SandAI-org/MagiAttention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

MoBA: Mixture of Block Attention for Long-Context LLMs

Trainable fast and memory-efficient sparse attention

Helpful tools and examples for working with flex-attention

Memory Sparse Attention - A scalable, end-to-end trainable latent-memory framework for 100M-token contexts.

CUDA kernels for linear attention variants, written in CuTe DSL and CUTLASS C++.

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

🚀 Efficient implementations for emerging model architectures

Fast and memory-efficient exact attention