apache/hamilton

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Scalable and efficient data transformation framework - backwards compatible with dbt.

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

visual data prep powered by python

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

An orchestration platform for the development, production, and observation of data assets.

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Lightweight and extensible compatibility layer between dataframe libraries!