NVIDIA-NeMo/Curator

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

Data and tools for generating and inspecting OLMo pre-training data.

A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

Create Custom LLMs

DataComp for Language Models

Optimized Agentic and LLM Bulk Processing Over Your Data