allenai/dolma

DataComp for Language Models

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

Scalable data pre processing and curation toolkit for LLMs

A powerful tool for creating datasets for LLM fine-tuning 、RAG and Eval

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

Create Custom LLMs

Curated list of datasets and tools for post-training.

⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡