AI prompts
base on ๐ง The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data. <h1 align="center">
<a
target="_blank"
href="https://mage.ai"
>
<img
align="center"
alt="Mage"
src="https://github.com/mage-ai/assets/blob/main/mascots/mascots-shorter.jpeg?raw=true"
style="width:100%;"
/>
</a>
</h1>
<p align="center">
๐ง A modern replacement for Airflow.
</p>
<p align="center">
<a
href="https://docs.mage.ai"
target="_blank"
><b>Documentation</b></a> ๐ช๏ธ
<a
href="https://youtu.be/GswOdShLGmg"
target="_blank"
><b>Get a 5 min overview</b></a> ๐
<a
href="https://demo.mage.ai"
target="_blank"
><b>Play with live tool</b></a> ๐ฅ
<a
href="https://www.mage.ai/chat"
target="_blank"
>
<b>Get instant help</b>
</a>
</p>
<div align="center">
<a
href="https://pypi.org/project/mage-ai/"
target="_blank"
>
<img alt="PyPi" src="https://img.shields.io/pypi/v/mage-ai?color=orange" />
</a>
<a
href="https://anaconda.org/conda-forge/mage-ai"
target="_blank"
>
<img src="https://anaconda.org/conda-forge/mage-ai/badges/version.svg" />
</a>
<a
href="https://opensource.org/licenses/Apache-2.0"
target="_blank"
>
<img alt="License" src="https://img.shields.io/github/license/mage-ai/mage-ai?color=red" />
</a>
<a
href="https://www.mage.ai/chat"
target="_blank"
>
<img alt="Slack" src="https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack" />
</a>
<a
href="https://github.com/mage-ai/mage-ai"
target="_blank"
>
<img alt="GitHub Stars" src="https://img.shields.io/github/stars/mage-ai/mage-ai?logo=github">
</a>
<a
href="https://hub.docker.com/r/mageai/mageai"
target="_blank"
>
<img alt="Docker pulls" src="https://img.shields.io/docker/pulls/mageai/mageai.svg">
</a>
<a
href="https://pepy.tech/project/mage-ai"
target="_blank"
>
<img alt="pip installs" src="https://static.pepy.tech/personalized-badge/mage-ai?period=total&units=international_system&left_color=grey&right_color=blue&left_text=pip%20installs">
</a>
</div>
<img
referrerpolicy="no-referrer-when-downgrade"
src="https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a"
/>
<div align="center">
### Give your data team `magical` powers
</div>
<p align="center">
<b>Integrate</b> and synchronize data from 3rd party sources
</p>
<p align="center">
Build real-time and batch pipelines to <b>transform</b> data using Python, SQL, and R
</p>
<p align="center">
Run, monitor, and <b>orchestrate</b> thousands of pipelines without losing sleep
</p>
<br />
<p align="center">1๏ธโฃ ๐๏ธ</p>
<h1 align="center">Build</h1>
<p align="center">
Have you met anyone who said they loved developing in Airflow?
<br />
Thatโs why we designed an easy developer experience that youโll enjoy.
</p>
| | |
| --- | --- |
| <b>Easy developer experience</b><br />Start developing locally with a single command or launch a dev environment in your cloud using Terraform.<br /><br/><b>Language of choice</b><br />Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.<br /><br /><b>Engineering best practices built-in</b><br />Each step in your pipeline is a standalone file containing modular code thatโs reusable and testable with data validations. No more DAGs with spaghetti code. | <img src="https://github.com/mage-ai/assets/blob/main/overview/mage-build.gif?raw=true" /> |
<p align="center">
โ
</p>
<p align="center">2๏ธโฃ ๐ฎ</p>
<h1 align="center">Preview</h1>
<p align="center">
Stop wasting time waiting around for your DAGs to finish testing.
<br />
Get instant feedback from your code each time you run it.
</p>
| | |
| --- | --- |
| <b>Interactive code</b><br />Immediately see results from your codeโs output with an interactive notebook UI.<br /><br/><b>Data is a first-class citizen</b><br />Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.<br /><br /><b>Collaborate on cloud</b><br />Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment. | <img src="https://github.com/mage-ai/assets/blob/main/overview/mage-preview.gif?raw=True" /> |
<p align="center">
โ
</p>
<p align="center">3๏ธโฃ ๐</p>
<h1 align="center">Launch</h1>
<p align="center">
Donโt have a large team dedicated to Airflow?
<br />
Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.
</p>
| | |
| --- | --- |
| <b>Fast deploy</b><br />Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.<br /><br/><b>Scaling made simple</b><br />Transform very large datasets directly in your data warehouse or through a native integration with Spark.<br /><br /><b>Observability</b><br />Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI. | <img src="https://github.com/mage-ai/assets/blob/main/overview/observability.gif?raw=True" /> |
<br />
# ๐ง Intro
Mage is an open-source data pipeline tool for transforming and integrating data.
1. [Install](#%EF%B8%8F-install)
1. [Demo](#-demo)
1. [Tutorials](#-tutorials)
1. [Documentation](https://docs.mage.ai)
1. [Features](#-features)
1. [Core design principles](https://docs.mage.ai/design/core-design-principles)
1. [Core abstractions](https://docs.mage.ai/design/core-abstractions)
1. [Contributing](https://docs.mage.ai/community/contributing)
<br />
# ๐โโ๏ธ Install
The recommended way to install the latest version of Mage is through Docker with the following command:
```bash
docker pull mageai/mageai:latest
```
You can also install Mage using pip or conda, though this may cause dependency issues without the proper environment.
```bash
pip install mage-ai
```
```bash
conda install -c conda-forge mage-ai
```
Looking for help? The _fastest_ way to get started is by checking out our documentation [here](https://docs.mage.ai/getting-started/setup).
Looking for quick examples? Open a [demo](https://demo.mage.ai/) project right in your browser or check out our [guides](https://docs.mage.ai/guides/overview).
# ๐ฎ Demo
### Live demo
Build and run a data pipeline with our <b>[demo app](https://demo.mage.ai/)</b>.
> WARNING
>
> The live demo is public to everyone, please donโt save anything sensitive (e.g. passwords, secrets, etc).
### Demo video (5 min)
[![Mage quick start demo](https://github.com/mage-ai/assets/blob/main/overview/overview-video.png?raw=True)](https://youtu.be/GswOdShLGmg)
<sub><i>Click the image to play video</i></sub>
<br />
# ๐ฉโ๐ซ Tutorials
- [Load data from API, transform it, and export it to PostgreSQL](https://docs.mage.ai/guides/load-api-data)
- [Integrate Mage into an existing Airflow project](https://docs.mage.ai/integrations/airflow)
- [Train model on Titanic dataset](https://docs.mage.ai/guides/train-model)
- [Set up dbt models and orchestrate dbt runs](https://docs.mage.ai/integrations/dbt-models)
<img alt="Fire mage" height="160" src="https://github.com/mage-ai/assets/blob/main/mage-fire-charging-up.svg?raw=True" />
<br />
# ๐ฎ [Features](https://docs.mage.ai/about/features)
| | | |
| --- | --- | --- |
| ๐ถ | <b>[Orchestration](https://docs.mage.ai/design/data-pipeline-management)</b> | Schedule and manage data pipelines with observability. |
| ๐ | <b>[Notebook](https://docs.mage.ai/about/features#notebook-for-building-data-pipelines)</b> | Interactive Python, SQL, & R editor for coding data pipelines. |
| ๐๏ธ | <b>[Data integrations](https://docs.mage.ai/data-integrations/overview)</b> | Synchronize data from 3rd party sources to your internal destinations. |
| ๐ฐ | <b>[Streaming pipelines](https://docs.mage.ai/guides/streaming-pipeline)</b> | Ingest and transform real-time data. |
| โ | <b>[dbt](https://docs.mage.ai/dbt/overview)</b> | Build, run, and manage your dbt models with Mage. |
<b>A sample data pipeline defined across 3 files โ</b>
1. Load data โ
```python
@data_loader
def load_csv_from_file():
return pd.read_csv('default_repo/titanic.csv')
```
1. Transform data โ
```python
@transformer
def select_columns_from_df(df, *args):
return df[['Age', 'Fare', 'Survived']]
```
1. Export data โ
```python
@data_exporter
def export_titanic_data_to_disk(df) -> None:
df.to_csv('default_repo/titanic_transformed.csv')
```
<b>What the data pipeline looks like in the UI โ</b>
<img
alt="data pipeline overview"
src="https://github.com/mage-ai/assets/blob/main/data-pipeline-overview.png?raw=True"
/>
New? We recommend reading about <b>[blocks](https://docs.mage.ai/design/blocks)</b> and
learning from a <b>[hands-on tutorial](https://docs.mage.ai/guides/load-api-data)</b>.
[![Ask us questions on Slack](https://img.shields.io/badge/%20-Ask%20us%20questions%20on%20Slack-purple?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)
<br />
# ๐๏ธ [Core design principles](https://docs.mage.ai/design/core-design-principles)
Every user experience and technical design decision adheres to these principles.
| | | |
| --- | --- | --- |
| ๐ป | [Easy developer experience](https://docs.mage.ai/design/core-design-principles#easy-developer-experience) | Open-source engine that comes with a custom notebook UI for building data pipelines. |
| ๐ข | [Engineering best practices built-in](https://docs.mage.ai/design/core-design-principles#engineering-best-practices-built-in) | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
| ๐ณ | [Data is a first-class citizen](https://docs.mage.ai/design/core-design-principles#data-is-a-first-class-citizen) | Designed from the ground up specifically for running data-intensive workflows. |
| ๐ช | [Scaling is made simple](https://docs.mage.ai/design/core-design-principles#scaling-is-made-simple) | Analyze and process large data quickly for rapid iteration. |
<br />
# ๐ธ [Core abstractions](https://docs.mage.ai/design/core-abstractions)
These are the fundamental concepts that Mage uses to operate.
| | |
| --- | --- |
| [Project](https://docs.mage.ai/design/core-abstractions#project) | Like a repository on GitHub; this is where you write all your code. |
| [Pipeline](https://docs.mage.ai/design/core-abstractions#pipeline) | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
| [Block](https://docs.mage.ai/design/core-abstractions#block) | A file with code that can be executed independently or within a pipeline. |
| [Data product](https://docs.mage.ai/design/core-abstractions#data-product) | Every block produces data after it's been executed. These are called data products in Mage. |
| [Trigger](https://docs.mage.ai/design/core-abstractions#trigger) | A set of instructions that determine when or how a pipeline should run. |
| [Run](https://docs.mage.ai/design/core-abstractions#run) | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
<br />
# ๐โโ๏ธ Contributing and developing
Add features and instantly improve the experience for everyone.
Check out the <b>[contributing guide](https://docs.mage.ai/community/contributing)</b>
to set up your development environment and start building.
<br />
# ๐จโ๐ฉโ๐งโ๐ฆ Community
Individually, weโre a mage.
> ๐ง Mage
>
> Magic is indistinguishable from advanced technology.
> A mage is someone who uses magic (aka advanced technology).
Together, weโre Magers!
> ๐งโโ๏ธ๐ง Magers (`/หmฤjษr/`)
>
> A group of mages who help each other realize their full potential!
Letโs hang out and chat together โ
[![Hang out on Slack](https://img.shields.io/badge/%20-Hang%20out%20on%20Slack-purple?style=for-the-badge&logo=slack&labelColor=6B50D7)](https://www.mage.ai/chat)
For real-time news, fun memes, data engineering topics, and more, join us on โ
| | |
| --- | --- |
| <img alt="Twitter" height="20" src="https://user-images.githubusercontent.com/78053898/198755056-a15c4439-c07f-41ea-ba35-bc4bfdd09f1a.png" /> | [Twitter](https://twitter.com/mage_ai) |
| <img alt="LinkedIn" height="20" src="https://user-images.githubusercontent.com/78053898/198755052-2777d6ae-c161-4a4b-9ece-4fd7bd458e26.png" /> | [LinkedIn](https://www.linkedin.com/company/magetech/mycompany) |
| <img alt="GitHub" height="20" src="https://user-images.githubusercontent.com/78053898/198755053-5c3971b1-9c49-4888-8a8e-1599f0fc6646.png" /> | [GitHub](https://github.com/mage-ai/mage-ai) |
| <img alt="Slack" height="20" src="https://user-images.githubusercontent.com/78053898/198755054-03d47bfc-18b6-45a5-9593-7b496eb927f3.png" /> | [Slack](https://www.mage.ai/chat) |
<br />
# ๐ค Frequently Asked Questions (FAQs)
Check out our [FAQ page](https://docs.mage.ai/about/frequently-asked-questions) to find answers to some of our most asked questions.
<br />
# ๐ชช License
See the [LICENSE](LICENSE) file for licensing information.
[<img alt="Water mage casting spell" height="300" src="https://github.com/mage-ai/assets/blob/main/mage-water-charging-up.svg?raw=True" />](https://www.mage.ai/)
<br />
", Assign "at most 3 tags" to the expected json: {"id":"7871","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"