This article explains the concept of Message Queue, What it is, How it works, and when to use it.

A Queue in this context could be defined as a list of data items, commands, or task, stored so as to be retrieved in a specific order, usually the order of insertion.

Message, on the other hand, is data transported between a sender and a receiver (e.g could be messages you sent to your friends on either WhatsApp or Facebook messenger).

But, in the context of our discussion the best definition we could give a message should be something that tells a system to start processing a particular task, it could contain information about a finished task or just a simple message of a completed task.

Now let's talk about message queue, A message queue is a form of asynchronous service-to-service communication used in microservices architectures (i.e a method of developing software systems that tries to focus on building single-function modules with well defined operations which could be written in any programming language). Messages are stored in the queue until they are processed and deleted. Each message is processed only once by a single consumer.

The basic structure or architecture of a message queue is simple, you have a client application called producers that create messages or make request to be delivered in a queue. Another application which is called a consumer, connect to the queue and get the messages to be processed. Messages placed in the queue could be stored in a system memory until the consumer processes them accordingly. The producer which is the client and the consumer of the message do not need to interact with the message queue synchronously.

Many producers and consumers can use the queue, but each message is processed only once by a single consumer. When a message needs to be processed by more than one consumer, message queues can be combined with Pub/Sub messaging in a fanout design pattern.

Let's talk about this in a use case scenario, imagine you have a web service or an API that receives many requests every second and all requests needs to be processed by a process that is time-consuming and your web service has to be highly available and ready to receive new request instead of being locked by the processing of previously received requests.

In this case, it is ideal to put a queue between the web service and the processing service. Messages in a queue and the other process can take and handle messages in order.

The two processes will be separated or decouple from each other and does not need to wait for each other. If you have a lot of requests coming in a short amount of time, the processing system will be able to process them all anyway. The queue will store or persist the requests if their number becomes really huge.

Now if your workload is growing and you need to scale up your system. All that is needed to be done now is to add more workers, receivers, to work off the queues faster.

Another use case could be when working on an imaging system where people upload images and then you have a service that generates thumbnails for each uploaded message, in this case, your best solution would be to implement the concept of “message queue”.

An example of a queue base solution is RabbitMQ, AXON, e.t.c