meta-pytorch/autoparallel

Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

Supercharge Your Model Training

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

PyTorch compiler that accelerates training and inference. Get built-in optimizations for performance, memory, parallelism, and easily write your own.

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

PyTorch Single Controller

JaxPP is a library for JAX that enables flexible MPMD pipeline parallelism for large-scale LLM training

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.