NVIDIA-NeMo/Emerging-Optimizers

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

Distributed Training Over-The-Internet

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Efficient Triton Kernels for LLM Training

Ongoing research training transformer models at scale

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

PyTorch native post-training library