facebookresearch/xformers

Fast and memory-efficient exact attention

Ongoing research training transformer models at scale

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

FlashMLA: Efficient Multi-head Latent Attention Kernels

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Mamba SSM architecture

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

Train transformer language models with reinforcement learning.