base on select * from logs; Tailpipe is an open source SIEM for instant log insights, powered by DuckDB. Analyze millions of events in seconds, right from your terminal. <a href="https://tailpipe.io"><img width="67%" src="https://tailpipe.io/images/pipeling-wordmarks/tailpipe_wordmark_white_outline.svg" /></a> [](https://hub.tailpipe-io.vercel.app/) &nbsp; [](https://hub.tailpipe-io.vercel.app/) &nbsp; [](https://turbot.com/community/join?utm_id=gspreadme&utm_source=github&utm_medium=repo&utm_campaign=github&utm_content=readme) &nbsp; [](https://turbot.com?utm_id=gspreadme&utm_source=github&utm_medium=repo&utm_campaign=github&utm_content=readme) # select * from logs; [Tailpipe](https://tailpipe.io) is the **lightweight**, **developer-friendly** way to query logs. **Cloud logs, SQL insights**. Collects logs from cloud, container and application sources. Query and analyze your data instantly with the power of SQL, right from your terminal. **Fast, local, and efficient**. Runs locally, powered by DuckDB's in-memory analytics and Parquet's optimized storage. **An ecosystem of prebuilt intelligence**. MITRE ATT&CK-aligned queries, prebuilt detections, benchmarks, and dashboards, all open source and community-driven. **Built to build with**. Define detections as code, extend functionality with plugins and write custom SQL queries. ## Demo time! **[Watch on YouTube →](https://www.youtube.com/watch?v=IR9MK1DMvW4)** <a href="https://www.youtube.com/watch?v=IR9MK1DMvW4"><img alt="tailpipe demo" width=500 src="https://tailpipe.io/images/video_preview.png"></a> ## Documentation See the [documentation](https://tailpipe.io/docs) for: - [Getting started](https://tailpipe.io/docs) - [It's just SQL!](https://tailpipe.io/docs/sql) - [Managing Tailpipe](https://tailpipe.io/docs/manage) - [CLI commands](https://tailpipe.io/docs/reference/cli) Plugins and query examples are on the [Tailpipe Hub](https://hub.tailpipe.io). Prebuilt detection benchmarks are on the [Powerpipe Hub](https://hub.powerpipe.io/?engines=tailpipe). ## Getting Started Install Tailpipe from the [downloads](https://tailpipe.io/downloads) page: ```sh # MacOS brew install turbot/tap/tailpipe ``` ```sh # Linux or Windows (WSL) sudo /bin/sh -c "$(curl -fsSL https://tailpipe.io/install/tailpipe.sh)" ``` Install a plugin from the [Tailpipe Hub](https://hub.tailpipe.io) for your favorite service (e.g. [AWS](https://hub.tailpipe.io/plugins/turbot/aws), [Azure](https://hub.tailpipe.io/plugins/turbot/azure), [GCP](https://hub.tailpipe.io/plugins/turbot/gcp)): ```sh tailpipe plugin install aws ``` Configure your [connection](https://tailpipe.io/docs/manage/connection) credentials, table [partition](https://tailpipe.io/docs/manage/partition) and data [source](https://tailpipe.io/docs/manage/source). Here is an [AWS CloudTrail example](https://hub.tailpipe.io/plugins/turbot/aws/tables/aws_cloudtrail_log#example-configurations): ```sh vi ~/.tailpipe/config/aws.tpc ``` ```hcl connection "aws" "logging_account" { profile = "my-logging-account" } partition "aws_cloudtrail_log" "my_logs" { source "aws_s3_bucket" { connection = connection.aws.logging_account bucket = "aws-cloudtrail-logs-bucket" } } ``` Download, enrich, and save logs from your source ([examples](https://tailpipe.io/docs/reference/cli/collect)): ```sh tailpipe collect aws_cloudtrail_log ``` Enter interactive query mode: ```sh tailpipe query ``` Run a query: ```sql select event_source, event_name, count(*) as event_count from aws_cloudtrail_log where not read_only group by event_source, event_name order by event_count desc; ``` ```sh +----------------------+-----------------------+-------------+ | event_source | event_name | event_count | +----------------------+-----------------------+-------------+ | logs.amazonaws.com | CreateLogStream | 793845 | | ecs.amazonaws.com | RunTask | 350836 | | ecs.amazonaws.com | SubmitTaskStateChange | 190185 | | s3.amazonaws.com | PutObject | 60842 | | sns.amazonaws.com | TagResource | 25499 | | lambda.amazonaws.com | TagResource | 20673 | +----------------------+-----------------------+-------------+ ``` ## Detections as Code with Powerpipe Pre-built dashboards and detections for the AWS plugin are available in [Powerpipe](https://powerpipe.io) mods, helping you monitor and analyze activity across your AWS accounts. For example, the [AWS CloudTrail Logs Detections mod](https://hub.powerpipe.io/mods/turbot/tailpipe-mod-aws-cloudtrail-log-detections) scans your CloudTrail logs for anomalies, such as an S3 bucket being made public or a change in your VPC network infrastructure. Dashboards and detections are [open source](https://github.com/topics/tailpipe-mod), allowing easy customization and collaboration. To get started, choose a mod from the [Powerpipe Hub](https://hub.powerpipe.io/?engines=tailpipe). ## Developing If you want to help develop the core Tailpipe binary, these are the steps to build it. **Clone**: ```sh git clone https://github.com/turbot/tailpipe ``` **Build**: ``` cd tailpipe make ``` **Check the version**: ``` $ tailpipe --version Tailpipe version 0.1.0 ``` ## Open source and contributing This repository is published under the [AGPL 3.0](https://www.gnu.org/licenses/agpl-3.0.html) license. Please see our [code of conduct](https://github.com/turbot/.github/blob/main/CODE_OF_CONDUCT.md). Contributors must sign our [Contributor License Agreement](https://turbot.com/open-source#cla) as part of their first pull request. We look forward to collaborating with you! [Tailpipe](https://tailpipe.io) is a product produced from this open source software, exclusively by [Turbot HQ, Inc](https://turbot.com). It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our [Open Source FAQ](https://turbot.com/open-source). ## Get involved **[Join #tailpipe on Slack →](https://turbot.com/community/join)** ", Assign "at most 3 tags" to the expected json: {"id":"12932","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"