In this article, we have covered the concept of Message queues in System Design in depth along with its properties, architecture, benefits, examples, message brokers and real life applications of Message Queues.

Table of contents:

1. What are message queues?
2. Properties of Message queues
3. Message Queue Architecture
4. How the architecture works?
5. Benefits of using message queues
6. Examples of Message Queues
7. Message brokers
8. Real-life application of Message Queues

Let us get started with Message Queues in System Design.

What are message queues?

A message queue is a messaging destination that uses the queue data structure to facilitate asynchronous communication between two services. They are commonly used in serverless and microservices architectures.
The message queue is comprised of two terms:

1. Message: this is the data that is passed from the producer to the consumer. The data can be requests, information, meta-data, etc.
2. Queue: this is a temporary buffer that stores messages. It uses the First-In-First-Out method to pass the messages from producer to consumer.

Properties of Message queues

• Message queues are asynchronous. This enables the producer to send a message and continue with other tasks without waiting for a response from the consumer. Once the consumer processes the message, it can then notify the producer.

• Message queues temporarily store messages until they are received and processed by the consumer.

• Each message is processed only once by a single consumer even though the message queue can be used by many producers and consumers.

Message Queue Architecture

The message queue architecture is made up of producers, consumers and the queue.

1. Producer: this is a client application that creates and places messages in the queue.

2. Consumer: this is the program or server that receives and processes the messages from the producer.

How the architecture works?

1. The producer creates the message and sends it to the message queue. In case the consumer is busy to immediately process it, the queue stores it until the consumer is available.

2. The consumer retrieves the message from the queue and starts processing it.

3. The message queue then temporarily locks the message to prevent it from being read by another consumer.

4. After the consumer completes the message processing, it deletes the message from the queue to prevent it from being read by other consumers.

Benefits of using message queues

• Message queues increase the reliability of systems as they persist data in case some part of the system goes offline.

• They improve performance as they enable asynchronous communication. This ensures that every system component is never idle waiting for a response or request.

• Message queues increase scalability as the system components are decoupled and thus can be independently scaled.

Examples of Message Queues

Some examples of message queues include Redis, Kafka, Heron, real-time streaming, Amazon SQS, etc. Redis is an in-memory data store that is usually used as either a high-performance key-value store or as a message broker. Redis Pub/Sub implements the messaging system where the publishers send the messages while the subscribers receive the messages. It uses a channel to transfer messages.

In most systems, you'll find message queues in message brokers. Message queues are components in message brokers that store messages until the consumers process the messages.

In the next section, we have covered the concept of message brokers in more detail.

Message brokers

A message broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. -Wikipedia.

Message brokers allow services and applications to communicate with each other using messages.

The message broker architecture is made up of the following:

• Producers: the applications that send messages.
• Consumers: receive and process the messages.
• Message queues: store the messages until they get processed.

One of the most popular message brokers is the RabbitMQ, which accepts and forwards messages.

In RabbitMQ, when the producer sends a message, it is not directly published to the queue, it is sent first to an exchange. The exchange is then responsible for routing messages to different queues depending on the binding and routing keys.
Binding is the link between a queue and the exchange.

The routing key is the key/address that the exchange uses to decide which queue to send the message.

Real-life application of Message Queues

Let's take a look at one use case of message queues in real-life applications. We have used Rabbit MQ to demonstrate this example.
In web applications that allow users to create pdf documents, the web applications are the producers, the PDF creator worker acts as the consumer, Rabbit MQ is the message broker.

In such a system, the process of creating a pdf entails:

• The user sends the request to create a pdf to the web application.
• The web application sends the message to Rabbit MQ.
• The Rabbit MQ exchange receives the message and routes it to the correct queue.
• The PDF processing worker then receives the message and creates the pdf.

With this article at OpenGenus, you must have a strong idea of what message queues are and how important they are when creating applications.