base on High throughput, Low latency 🚀 Fullstack Realtime messaging platform built using Go and React ❤️ # Real-time Distributed Chat This repo contains the **frontend** (React.js), **backend** (Go-Fiber) and **Infrastructure** (Terraform, CI/CD) code for building a distributed, real-time, scalable messaging platform. If you are a developer seeking to learn system design or even looking to find how end-to-end projects are build, I hope you find this useful ❤️ <br /> I will be writing articles on multiple features about the project like Configuring nginx as a reverse proxy for loadbalancing, TLS/SSL certificate for HTTPS communication, Setting up infrastructure using Terraform etc. So do follow me on HashNode (https://joyalajohney.hashnode.dev/). if you like the repo, please consider giving it a ⭐! . <img src="https://raw.githubusercontent.com/JoyalAJohney/Realtime-Distributed-Chat/main/assets/home.png" alt="landing page"> <div align="center"> <br /> <img src="https://img.shields.io/badge/React-20232A?style=for-the-badge&logo=react&logoColor=61DAFB" alt="React Badge"> * <img src="https://img.shields.io/badge/nginx-%23009639.svg?style=for-the-badge&logo=nginx&logoColor=white" alt="Nginx Badge"> * <img src="https://img.shields.io/badge/Go-00ADD8?style=for-the-badge&logo=go&logoColor=white" alt="Golang Badge"> * <img src="https://img.shields.io/badge/redis-%23DD0031.svg?&style=for-the-badge&logo=redis&logoColor=white" alt="Redis Badge"> * <img src="https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white" alt="Postgres Badge"> * <img src="https://img.shields.io/badge/Apache%20Kafka-000?style=for-the-badge&logo=apachekafka" alt="Kafka Badge"> * <img src="https://img.shields.io/badge/terraform-%235835CC.svg?style=for-the-badge&logo=terraform&logoColor=white" alt="Terraform Badge"> * <img src="https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white" alt="Docker Badge"> </div> ## Architecture Diagram 🌠 <img src="https://raw.githubusercontent.com/JoyalAJohney/Realtime-Distributed-Chat/main/assets/diagrams_image.png" alt="landing page"> ## Product Demo 🚀 https://github.com/JoyalAJohney/Realtime-Distributed-Chat/assets/31545426/db55bf32-1e35-4071-a80e-9f4944614e71 ## About the Project 🌌 * Multiple Go-fiber servers providing API endpoints (JWT authentication) and WebSocket connections for full-duplex communication. These Go instances are configured under Nginx (reverse proxy) Which act a layer 7 loadbalancer. * To propagate messages for users within the same room but connected to multiple instances, we utilize Redis (Pub/Sub model). Each instance is subscribed to a particular channel in Redis and gets notified on receiving messages. All messages are stored in Postgres. * The database can undergo a heavy write load if we receive 100 messages/sec. To avoid this, we use kafka, a message stream designed for high throughput and low latency processing. A consumer (Go instance) will consume messages from kafka in batches and writes them to postgres. * The frontend for application is build using React.js and served in an Nginx container. All the nodes are containarized using Docker and Configured using Docker-Compose. We only expose the Reverse-Proxy (Nginx) to the outside world. Al requests are redirected from there. * Next step is to deploy the application on AWS. A CI/CD pipeline is implemented using github actions. We use Terraform for setting up Infrastructure on AWS, configuring an EC2 instance, S3 storage, Security groups and Elastic IP. This is ntegrated into the CI/CD pipeline. * For secure HTTPS access, Issue certificate and configure it in Nginx for secure TLS/SSL communication. ## Setting Up 🔧 * Create a .env file from the env.sample file. * Fill in the values based on your required configuration. * Make sure that the .env file is in the same level as docker-compose.yml file ```bash # Redis Config REDIS_PORT=6379 REDIS_HOST=redis # Database Config POSTGRES_DATABASE=chat_db POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres POSTGRES_HOST=postgres POSTGRES_PORT=5432 # Kafka config KAFKA_HOST=kafka KAFKA_PORT=9092 KAFKA_TOPIC=chat_messages KAFKA_GROUP_ID=chat_group ZOOKEEPER_PORT=2181 # Authentication JWT_SECRET=secret # Reverse proxy config NGINX_ENV=local NGINX_PORT=8080 NGINX_HOST=localhost # LLM config LLM_PORT=11434 # Backend Servers SERVER_PORT=8080 ``` I have only shared the template code for **Infrastructre** (pipeline & terraform). You need to configure the AWS credentials, Terraform, and Pipeline according to your requirements. Additionally, you'll have to set up TLS/SSL for HTTPS (since all of this information is sensitive, it has been omitted) ## Running the app if you wish to run llm model, uncomment the changes from docker-compose (llama 2 model requires almost 3.6GB size) Execute the below command to build the application containers ```bash $ docker-compose up --build ``` If the application starts perfectly fine, you should be able to head over to http://localhost:8080/ ", Assign "at most 3 tags" to the expected json: {"id":"7003","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"