togethercomputer/RedPajama-Data

Data and tools for generating and inspecting OLMo pre-training data.

DataComp for Language Models

A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

Curated list of datasets and tools for post-training.

Scalable data pre processing and curation toolkit for LLMs

AllenAI's post-training codebase

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools