base on Apache Pulsar - distributed pub-sub messaging system <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->  [](https://hub.docker.com/r/apachepulsar/pulsar) [](https://github.com/apache/pulsar/graphs/contributors) [](https://github.com/apache/pulsar/commits/master) [](https://pulsar.apache.org/download/) [](https://pulsar.apache.org/download/) Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. Learn more about Pulsar at https://pulsar.apache.org ## Main features - Horizontally scalable (Millions of independent topics and millions of messages published per second) - Strong ordering and consistency guarantees - Low latency durable storage - Topic and queue semantics - Load balancer - Designed for being deployed as a hosted service: - Multi-tenant - Authentication - Authorization - Quotas - Support mixing very different workloads - Optional hardware isolation - Keeps track of consumer cursor position - REST API for provisioning, admin and stats - Geo replication - Transparent handling of partitioned topics - Transparent batching of messages ## Repositories This repository is the main repository of Apache Pulsar. Pulsar PMC also maintains other repositories for components in the Pulsar ecosystem, including connectors, adapters, and other language clients. - [Pulsar Core](https://github.com/apache/pulsar) ### Helm Chart - [Pulsar Helm Chart](https://github.com/apache/pulsar-helm-chart) ### Ecosystem - [Pulsar Adapters](https://github.com/apache/pulsar-adapters) ### Clients - [.NET/C# Client](https://github.com/apache/pulsar-dotpulsar) - [C++ Client](https://github.com/apache/pulsar-client-cpp) - [Go Client](https://github.com/apache/pulsar-client-go) - [NodeJS Client](https://github.com/apache/pulsar-client-node) - [Python Client](https://github.com/apache/pulsar-client-python) - [Reactive Java Client](https://github.com/apache/pulsar-client-reactive) ### Dashboard & Management Tools - [Pulsar Manager](https://github.com/apache/pulsar-manager) ### Website - [Pulsar Site](https://github.com/apache/pulsar-site) ### CI/CD - [Pulsar CI](https://github.com/apache/pulsar-test-infra) ### Archived/Halted - [Pulsar Connectors](https://github.com/apache/pulsar-connectors) (bundled as [pulsar-io](pulsar-io)) - [Pulsar Translation](https://github.com/apache/pulsar-translation) - [Pulsar SQL (Pulsar Presto Connector)](https://github.com/apache/pulsar-presto) (bundled as [pulsar-sql](pulsar-sql)) - [Ruby Client](https://github.com/apache/pulsar-client-ruby) ## Pulsar Runtime Java Version Recommendation > **Note**: > > When using Java versions, it is recommended to > use [a recent version of a particular Java release (17 or 21)](https://adoptium.net/en-GB/temurin/releases?version=21&os=linux&arch=any) > with the most recent bug fixes and security patches. > For example, the JVM bug [JDK-8351933](https://bugs.openjdk.org/browse/JDK-8351933) can cause stability issues in > Pulsar. > [JDK-8351933](https://bugs.openjdk.org/browse/JDK-8351933) was fixed in Java 17.0.17+ and Java 21.0.8+. > Pulsar Docker images come with the most recent Java version at the time of release. When using the Pulsar Java client, it is recommended to [set specific system properties and JVM options](https://pulsar.apache.org/docs/next/client-libraries-java-setup/#java-client-performance) to allow optimal performance. ### pulsar ver >= 4.1 and master branch | Component | Java Version | |-----------------|:------------:| | Broker | 21 | | Functions / IO | 21 | | CLI | 17 or 21 | | Java Client | 17 or 21 | Docker image Java runtime: 21 ### 3.3 <= pulsar ver <= 4.0 | Component | Java Version | |-----------------|:---------------:| | Broker | 21 | | Functions / IO | 21 | | CLI | 17 or 21 | | Java Client | 8, 11, 17 or 21 | Docker image Java runtime: 21 ### 2.10 <= pulsar ver <= 3.0 | Component | Java Version | |----------------|:-------------:| | Broker | 17 | | Functions / IO | 17 | | CLI | 17 | | Java Client | 8 or 11 or 17 | Docker image Java runtime: 17 ### 2.8 <= pulsar ver <= 2.10 | Component | Java Version | |-----------------|:------------:| | Broker | 11 | | Functions / IO | 11 | | CLI | 8 or 11 | | Java Client | 8 or 11 | ### pulsar ver < 2.8 | Component | Java Version | |-------------|:------------:| | All | 8 or 11 | ## Build Pulsar ### Requirements - JDK | Pulsar Version | JDK Version | |------------------|:--------------------------------------------------------------------------------:| | master and 4.0+ | [JDK 21](https://adoptium.net/en-GB/temurin/releases?version=21&os=any&arch=any) | | 2.11 + | [JDK 17](https://adoptium.net/en-GB/temurin/releases?version=17&os=any&arch=any) | | 2.8 / 2.9 / 2.10 | [JDK 11](https://adoptium.net/en-GB/temurin/releases?version=11&os=any&arch=any) | | 2.7 - | [JDK 8](https://adoptium.net/en-GB/temurin/releases?version=8&os=any&arch=any) | - Maven 3.9.9+ - zip There is also a guide for [setting up the tooling for building Pulsar](https://pulsar.apache.org/contribute/setup-buildtools/). > **Note**: > > This project includes a [Maven Wrapper](https://maven.apache.org/wrapper/) that can be used instead of a system-installed Maven. > Use it by replacing `mvn` by `./mvnw` on Linux and `mvnw.cmd` on Windows in the commands below. ### Build Compile and install: ```bash $ mvn install -DskipTests ``` Compile and install individual module ```bash $ mvn -pl module-name (e.g: pulsar-broker) install -DskipTests ``` ### Minimal build (This skips most of external connectors and tiered storage handlers) ```bash mvn install -Pcore-modules,-main -DskipTests ``` Run Unit Tests: ```bash $ mvn test ``` Run Individual Unit Test: ```bash $ mvn -pl module-name (e.g: pulsar-client) test -Dtest=unit-test-name (e.g: ConsumerBuilderImplTest) ``` Run Selected Test packages: ```bash $ mvn test -pl module-name (for example, pulsar-broker) -Dinclude=org/apache/pulsar/**/*.java ``` Start standalone Pulsar service: ```bash $ bin/pulsar standalone ``` Check https://pulsar.apache.org for documentation and examples. ## Build custom docker images The commands used in the Apache Pulsar release process can be found in the [release process documentation](https://pulsar.apache.org/contribute/release-process/#stage-docker-images). Here are some general instructions for building custom docker images: * Docker images must be built with Java 8 for `branch-2.7` or previous branches because of [ISSUE-8445](https://github.com/apache/pulsar/issues/8445). * Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`. * Java 17 is the recommended JDK version in `branch-2.11`, `branch-3.0` and `branch-3.3`. * Java 21 is the recommended JDK version since `branch-4.0`. The following command builds the docker images `apachepulsar/pulsar-all:latest` and `apachepulsar/pulsar:latest`: ```bash mvn clean install -DskipTests # setting DOCKER_CLI_EXPERIMENTAL=enabled is required in some environments with older docker versions export DOCKER_CLI_EXPERIMENTAL=enabled mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests ``` After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to `localhost:32000/apachepulsar`. ```bash image_repo_and_project=localhost:32000/apachepulsar pulsar_version=$(mvn initialize help:evaluate -Dexpression=project.version -pl . -q -DforceStdout) gitrev=$(git rev-parse HEAD | colrm 10) tag="${pulsar_version}-${gitrev}" echo "Using tag $tag" docker tag apachepulsar/pulsar-all:latest ${image_repo_and_project}/pulsar-all:$tag docker push ${image_repo_and_project}/pulsar-all:$tag docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag docker push ${image_repo_and_project}/pulsar:$tag ``` ## Setting up your IDE Read https://pulsar.apache.org/contribute/setup-ide for setting up IntelliJ IDEA or Eclipse for developing Pulsar. ## Documentation > **Note**: > > For how to make contributions to Pulsar documentation, see [Pulsar Documentation Contribution Guide](https://pulsar.apache.org/contribute/document-intro/). ## Contact ##### Mailing lists * The mailing lists are the primary contact for the Apache Pulsar project. * Your email to the mailing list might be placed in a moderation queue if you haven't joined the mailing list by subscribing before posting. > **Note** > > Please note that security-related issues or concerns should not be reported in public channels. > Follow the instructions in the [Security Policy](https://pulsar.apache.org/security/) to contact the [ASF Security Team](https://www.apache.org/security/) and the Apache Pulsar PMC directly. | Name | Scope | Subscribe | Unsubscribe | Archives | |:----------------------------------------------------------|:--------------------------------|:------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------| | [[email protected]](mailto:[email protected]) | User-related discussions | [Subscribe](mailto:[email protected]?subject=subscribe&body=subscribe) | [Unsubscribe](mailto:[email protected]?subject=unsubscribe&body=unsubscribe) | [Archives](https://lists.apache.org/[email protected]) | | [[email protected]](mailto:[email protected]) | Development-related discussions | [Subscribe](mailto:[email protected]?subject=subscribe&body=subscribe) | [Unsubscribe](mailto:[email protected]?subject=unsubscribe&body=unsubscribe) | [Archives](https://lists.apache.org/[email protected]) | ##### Slack Pulsar slack channel at https://apache-pulsar.slack.com/ You can self-register at https://communityinviter.com/apps/apache-pulsar/apache-pulsar ## Security Policy If you find a security issue with Pulsar then please [read the security policy](https://pulsar.apache.org/security/#security-policy). It is critical to avoid public disclosure. ### Reporting a security vulnerability To report a vulnerability for Pulsar, contact the [Apache Security Team](https://www.apache.org/security/). When reporting a vulnerability to [[email protected]](mailto:[email protected]), you can copy your email to [[email protected]](mailto:[email protected]) to send your report to the Apache Pulsar Project Management Committee. This is a private mailing list. https://github.com/apache/pulsar/security/policy contains more details. ## License Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 ## Crypto Notice This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See [The Wassenaar Arrangement](http://www.wassenaar.org/) for more information. The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code. The following provides more details on the included cryptographic software: Pulsar uses the SSL library from Bouncy Castle written by http://www.bouncycastle.org. ", Assign "at most 3 tags" to the expected json: {"id":"1576","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"