pytorch/gloo

Optimized primitives for collective multi-GPU communication

UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

A fast communication-overlapping library for tensor/expert parallelism on GPUs.

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

DeepEP: an efficient expert-parallel communication library

Supercharge Your Model Training

Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

Machine learning metrics for distributed, scalable PyTorch applications.

NCCL Tests