drisspg/transformer_nuggets

A collection of tricks and tools to speed up transformer models

Hackable and optimized Transformers building blocks, supporting a composable construction.

Utils for Unsloth https://github.com/unslothai/unsloth

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Ongoing research training transformer models at scale

A Quirky Assortment of CuTe Kernels

A concise but complete full-attention transformer with a set of promising experimental features from various papers

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

Prune transformer layers