This is a demonstration of how to create simple yet capable architecture with MOM (Message-Oriented-Middleware) approach to achieve well scalable and distributed systems. On the tip of the iceberg we're only seeing an admin panel, where you can register, login, order and list jobs etc. But under the hood (or below the sea level?) whole app running by queuing messages to do things.
Here is a diagram to visualise how our architecture works.
- Core Concept
- Additional Pieces
- Installation and Running
- Technologies Used
- Management Screens
- Connection Points
- Using Sandbox
- Debugging
- Resources for Better Understanding
The core concept is; when there is a work which need to be done; services wont communicate to related service directly. Instead of that they will leave a message to message queue, which will be consumed by service (worker) that responsible of that work.
For example, in traditional systems when an user want to register, he or she calls the api and api will create a new db record and return some response to user. But here, API doesn't create db record itself, instead of that; it leaves a message to message queue which have details about what should be done (and data like email, password etc.) and waits for some db-worker to take care of it.
This waiting technique actually named RPC and it means that message leaver will wait until job is done, and will do something with returned data (like informing user directly by returning a response). We're using RPC only in jobs like user register or job creation for make an example. RPC messages will have top priority (10) which cause them to be processed by worker before ordinary (lower priority) messages.
On the other hand, there are jobs which are not in a hurry and needs to be done sometime. Like math tasks. There is a screen in panel where you can schedule math tasks. Like sum or multiply these numbers etc. And there is a math-worker inside worker service which will call another external math service to get results of this calculations. And to inform user after these kind of deferred works, we're using socket connections. (Our math service will respond between 0~10 second delay for demonstration purposes.)
But what if user not in panel and there is no socket connection? Well, after a deferred task, beside trying to inform user over socket, of course we should also persist result of that job into database. (By leaving a message to message queue for db-worker.) So even user not in panel, he/she will see information on next login. (I didnt implement this, but this is the idea.)
Seperated from our main application; the purpose of sandbox is making experiments for grasp better understanding over concepts, libraries etc. For more information and to learn how to use sandbox see using sandbox section.
Some bash scripts to use when needed.
To install and run application you only need to have installed and running docker.
git clone git@github.com:ramesaliyev/mom.git
docker-compose up --build
Starting everything will take some time, so be patient. When everything has started you can;
- Navigate to admin panel at localhost:9090
- See other management screens
docker-compose up postgresql_server postgresql_adminer rabbitmq_server redis_server redis_commander
docker-compose stop <servicename>
# example
docker-compose stop api
docker-compose up --build <servicename>
# example
docker-compose up --build api
You can start every service outside of docker.
1- Navigate to service folder.
cd services/be-api
2- Install dependencies.
npm i
3- Start in development mode.
npm start
4- Or Production mode.
npm run start:prod
Notices:
- In docker every service starts in production mode by default.
- There is no watch mode in development. So you need to restart a service if you change something.
- You need to rebuild docker image if you want to start a service in docker after you change something.
- Backend
- Frontend
- Our Panel: localhost:9090
- PostgreSQL: localhost:9091
- system: PostgreSQL
- server: postgresql_server
- username: poc_db_user
- password: poc_db_pass
- database: poc_db_name
- RabbitMQ: localhost:9092
- username: poc_rmq_user
- password: poc_rmq_pass
- Redis: localhost:9093
- API:
- hostname: localhost:7070
- Socket:
- hostname: localhost:7080
- External Math Service API:
- hostname: localhost:7090
- PostgreSQL
- hostname: localhost:7071
- others same as above
- RabbitMQ
- hostname: localhost:7072
- others same as above
- Redis
- hostname: localhost:7073
There is a sandbox to ease making experiments with things. It consist of mostly little scripts written in plain javascript by following tutorials etc. You can add something under the sandbox and run it within container. Since folder already added as volume you dont need to restart anything. To enter in sandbox;
docker exec -it poc_sandbox /bin/sh
And when you in, simply run scripts as you want;
node rabbitmq/tutorial/hello-world/send.js
Create a new terminal window and run the receiver also;
node rabbitmq/tutorial/hello-world/receive.js
You can run multiple receivers to observe things.
These debug instructions are for VSCode only.
-
Install latest version of VS Code and VS Code Chrome Debugger Extension.
-
Start your app (outside of the docker) by running
npm start. -
Now start debugging in VS Code by choosing correct configuration (ex:
Fe/Panel). -
You can now write code, set breakpoints, make changes to the code, and debug your newly modified code, all from your editor.
-
Navigate to
.tsfile you would want to debug. -
Now start debugging in VS Code by choosing correct configuration (ex:
Be/Api). -
You can now write code, set breakpoints, make changes to the code, and debug your newly modified code, all from your editor.
For example: you want to debug api, navigate to src/modules/auth/auth.controller.ts and add breakpoint to login method. Then in postman or from frontend, trigger the api.
- Docker
- MOM
- RabbitMQ
- Dictionary
- Official
- Other
- TypeScript