base on Stream your Postgres data anywhere in real-time. Simple Rust building blocks for change data capture (CDC) pipelines. <br /> <p align="center"> <a href="https://supabase.com"> <picture> <img alt="ETL by Supabase" width="100%" src="docs/assets/etl-logo-extended.png"> </picture> </a> <h1 align="center">ETL</h1> <p align="center"> <a href="https://github.com/supabase/etl/actions/workflows/ci.yml"> <img alt="CI" src="https://github.com/supabase/etl/actions/workflows/ci.yml/badge.svg?branch=main"> </a> <a href="https://coveralls.io/github/supabase/etl?branch=main"> <img alt="Coverage Status" src="https://coveralls.io/repos/github/supabase/etl/badge.svg?branch=main"> </a> <a href="https://github.com/supabase/etl/actions/workflows/docs.yml"> <img alt="Docs" src="https://github.com/supabase/etl/actions/workflows/docs.yml/badge.svg?branch=main"> </a> <a href="https://github.com/supabase/etl/actions/workflows/docker-build.yml"> <img alt="Docker Build" src="https://github.com/supabase/etl/actions/workflows/docker-build.yml/badge.svg?branch=main"> </a> <a href="https://github.com/supabase/etl/actions/workflows/audit.yml"> <img alt="Security Audit" src="https://github.com/supabase/etl/actions/workflows/audit.yml/badge.svg?branch=main"> </a> <a href="LICENSE"> <img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-blue.svg"> </a> <br /> Build real-time Postgres replication applications in Rust <br /> <a href="https://supabase.github.io/etl"><strong>Documentation</strong></a> · <a href="https://github.com/supabase/etl/tree/main/etl-examples"><strong>Examples</strong></a> · <a href="https://github.com/supabase/etl/issues"><strong>Issues</strong></a> </p> </p> ETL is a Rust framework by [Supabase](https://supabase.com) for building high‑performance, real‑time data replication apps on Postgres. It sits on top of Postgres [logical replication](https://www.postgresql.org/docs/current/protocol-logical-replication.html) and gives you a clean, Rust‑native API for streaming changes to your own destinations. ## Highlights - 🚀 Real‑time replication: stream changes as they happen - ⚡ High performance: batching and parallel workers - 🛡️ Fault tolerant: retries and recovery built in - 🔧 Extensible: implement custom stores and destinations - 🧭 Typed, ergonomic Rust API ## Get Started Install via Git while we prepare for a crates.io release: ```toml [dependencies] etl = { git = "https://github.com/supabase/etl" } ``` Quick example using the in‑memory destination: ```rust use etl::{ config::{BatchConfig, PgConnectionConfig, PipelineConfig, TlsConfig}, destination::memory::MemoryDestination, pipeline::Pipeline, store::both::memory::MemoryStore, }; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let pg = PgConnectionConfig { host: "localhost".into(), port: 5432, name: "mydb".into(), username: "postgres".into(), password: Some("password".into()), tls: TlsConfig { enabled: false, trusted_root_certs: String::new() }, }; let store = MemoryStore::new(); let destination = MemoryDestination::new(); let config = PipelineConfig { id: 1, publication_name: "my_publication".into(), pg_connection: pg, batch: BatchConfig { max_size: 1000, max_fill_ms: 5000 }, table_error_retry_delay_ms: 10_000, max_table_sync_workers: 4, }; let mut pipeline = Pipeline::new(config, store, destination); pipeline.start().await?; // pipeline.wait().await?; // Optional: block until completion Ok(()) } ``` For tutorials and deeper guidance, see the [Documentation](https://supabase.github.io/etl) or jump into the [examples](etl-examples/README.md). ## Destinations ETL is designed to be extensible. You can implement your own destinations to send data to any destination you like, however it comes with a few built in destinations: - BigQuery Out-of-the-box destinations are available in the `etl-destinations` crate: ```toml [dependencies] etl = { git = "https://github.com/supabase/etl" } etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] } ``` ## License Apache‑2.0. See `LICENSE` for details. --- <p align="center"> Made with ❤️ by the <a href="https://supabase.com">Supabase</a> team </p> ", Assign "at most 3 tags" to the expected json: {"id":"11449","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"