google-research/deduplicate-text-datasets

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

Data and tools for generating and inspecting OLMo pre-training data.

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

DataComp for Language Models

Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"

Scalable data pre processing and curation toolkit for LLMs

Code for the paper "Language Models are Unsupervised Multitask Learners"

A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

Language model tokenization at GB/s