feifeibear/long-context-attention

A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training

Trainable fast and memory-efficient sparse attention

MoBA: Mixture of Block Attention for Long-Context LLMs

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

🚀 Efficient implementations for emerging model architectures

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

CUDA kernels for linear attention variants, written in CuTe DSL and CUTLASS C++.

Fast and memory-efficient exact attention

Helpful tools and examples for working with flex-attention