Lightning-AI/litData

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.

Efficient framework-agnostic data loading

Library for reading and processing ML training data.

Task-based datasets, preprocessing, and evaluation for sequence models.

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

Datasets, Transforms and Models specific to Computer Vision

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

Simple, safe way to store and distribute tensors

Scalable data pre processing and curation toolkit for LLMs