r-three/common-pile

Let's build better datasets, together!

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

Scalable data pre processing and curation toolkit for LLMs

Datasets collection and preprocessings framework for NLP extreme multitask learning

A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

Quickly download, clean up, and install public datasets into a database management system

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.