nntile/nntile

YaFSDP: Yet another Fully Sharded Data Parallel

Making large AI models cheaper, faster and more accessible

RAM is all you need

FlexTensor is a tensor offloading and management library for PyTorch that enables running large models on limited GPU memory by intelligently offloading tensors between GPU and CPU memory.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A fast communication-overlapping library for tensor/expert parallelism on GPUs.

Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

Fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters