meta-pytorch/attention-gym

Fast and memory-efficient exact attention

🚀 Efficient implementations for emerging model architectures

Hackable and optimized Transformers building blocks, supporting a composable construction.

A concise but complete full-attention transformer with a set of promising experimental features from various papers

FlashMLA: Efficient Multi-head Latent Attention Kernels

Provide with pre-build flash-attention 2 and 3 package wheels on Linux and Windows using GitHub Actions

FlashKDA: high-performance Kimi Delta Attention kernels

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support