olirice/flupy

Flowman is an ETL framework powered by Apache Spark. With its declarative approach, Flowman simplifies the development of complex data pipelines.

sync/async iterable streams for Python

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Turning PySpark Into a Universal DataFrame API

A simplified, lightweight ETL Framework based on Apache Spark

The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️