yandex/YaFSDP

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

Making large AI models cheaper, faster and more accessible

Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

Fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters

Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Distributed Training Over-The-Internet

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training