base on eBPF Observability - Distributed Tracing and Profiling <p align="center"> <img src="./docs/deepflow-logo.png" alt="DeepFlow" width="300" /> <p align="center">Instant Observability for Cloud & AI Applications</p> <p align="center">Zero Code, Full Stack, eBPF & Wasm</p> </p> <p align="center"> <a href="https://zenodo.org/badge/latestdoi/448599559"><img src="https://zenodo.org/badge/448599559.svg" alt="DOI"></a> <img alt="GitHub Release" src="https://img.shields.io/github/v/release/deepflowio/deepflow"> </a> <img alt="docker pulls" src="https://img.shields.io/docker/pulls/deepflowce/deepflow-agent?color=green?label=docker pulls"> </a> <img alt="License" src="https://img.shields.io/github/license/deepflowio/deepflow?color=purple"> </a> </p> ------------- English | [简体中文](./README-CN.md) | [日本語](./README-JP.md) # What is DeepFlow The DeepFlow open-source project aims to provide deep observability for complex cloud-native and AI applications. DeepFlow implemented **Zero Code** data collection with eBPF for metrics, distributed tracing, request logs and function profiling, and is further integrated with **SmartEncoding** to achieve **Full Stack** correlation and efficient access to all observability data. With DeepFlow, cloud-native and AI applications automatically gain deep observability, removing the heavy burden of developers continually instrumenting code and providing monitoring and diagnostic capabilities covering everything from code to infrastructure for DevOps/SRE teams. # Key Features - **Universal Map** for **Any** Service: DeepFlow provides a universal map with **Zero Code** by eBPF for production environments, including application services, AI services, and infrastructure services in any language. In addition to analyzing common protocols, Wasm plugins are supported for your private protocols. **Full Stack** golden signals of applications and infrastructures are calculated, pinpointing performance bottlenecks at ease. - **Distributed Tracing** for **Any** Request: **Zero Code** distributed tracing powered by eBPF supports applications in any language and infrastructures including gateways, service meshes, databases, message queues, DNS and NICs, leaving no blind spots. **Full Stack** network performance metrics and file I/O events are automatically collected for each Span. Distributed tracing enters a new era: Zero Instrumentation. - **Continuous Profiling** for **Any** Function: DeepFlow collects profiling data at a cost of below 1% with **Zero Code**, plots OnCPU/OffCPU/GPU/Memory/Network function call stack flame graphs, locates **Full Stack** performance bottleneck in business functions, library and framework functions, runtime functions, shared library functions, kernel function, CUDA functions, and automatically relates them to distrubuted tracing data. - **Seamless Integration** with Popular Stack: DeepFlow can serve as storage backed for Prometheus, OpenTelemetry, SkyWalking and Pyroscope. It also provides **SQL, PromQL and OLTP** APIs to work as data source in popular observability stacks. It injects meta tags for all observability signals including cloud resource, K8s container, K8s labels, K8s annotations, CMDB business attributes, etc., eliminating data silos. - **Performance 10x ClickHouse**: **SmartEncoding** injects standardized and pre-encoded meta tags into all observability data, reducing storage overhead by 10x compared to ClickHouse String or LowCard method. Custom tags and observability data are stored separately, making tags available for almost unlimited dimensions and cardinalities with uncompromised query experience like **BigTable**. # Documentation For more information, please visit [the documentation website](https://deepflow.io/docs/?from=github). # Quick start There are three editions of DeepFlow: - DeepFlow Community: for developers - DeepFlow Enterprise: for organizations, solving team collaboration problems - DeepFlow Cloud: SaaS service, currently in beta The DeepFlow Community Edition consists of the core components of the Enterprise Edition. ## DeepFlow Community Please refer to [the deployment documentation](https://deepflow.io/docs/ce-install/all-in-one/?from=github). At the same time, we have also built a complete [DeepFlow Community Demo](https://ce-demo.deepflow.yunshan.net/?from=github), welcome to experience it. Login account/password: deepflow/deepflow. ## DeepFlow Enterprise You can visit the [DeepFlow Enterprise Demo](https://deepflow.io/), currently available in Chinese only. # Compile DeepFlow from Source - [compile deepflow-agent](./agent/build.md) # Software Architecture DeepFlow Community Edition consists of two components, Agent and Server. An Agent runs in each K8s node, legacy host and cloud host, and is responsible for AutoMetrics and AutoTracing data collection of all application processes on the host. Server runs in a K8s cluster and provides Agent management, tag injection, data ingest and query services.  # Milestones Here is our [future feature plan](https://deepflow.io/docs/about/milestone/?from=github). Issues and Pull Requests are welcome. # Contact Us - Discord：Click [here](https://discord.gg/QJ7Dyj4wWM) to join our discussion. - Twitter：[DeepFlow](https://twitter.com/deepflowio) - WeChat Group： <img src=./docs/wechat-group-keeper.png width=30% /> # Acknowledgments - Thanks [eBPF](https://ebpf.io/), a revolutionary Linux kernel technology. - Thanks [OpenTelemetry](https://opentelemetry.io/), provides vendor-neutral APIs to collect application telemetry data. # Honors - The paper [Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero Code](https://dl.acm.org/doi/10.1145/3603269.3604823) has been accepted by ACM SIGCOMM 2023. - DeepFlow enriches the <a href="https://landscape.cncf.io/?selected=deep-flow">CNCF CLOUD NATIVE Landscape</a>. - DeepFlow enriches the <a href="https://landscape.cncf.io/?selected=deep-flow&group=cnai&item=cnai--model-llm-observability--deepflow">CNCF CNAI (Cloud-Native AI) Landscape</a>. - DeepFlow enriches the <a href="https://ebpf.io/applications#deepflow">eBPF Project Landscape</a>. ", Assign "at most 3 tags" to the expected json: {"id":"6832","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"