Message Queues and Multiple Subscribers

Question

My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at this time (likely through Boomi or Idocs). We know next to nothing about RabbitMQ, except the general concepts of a message queue and what we've been able to determine from a high level look at their documentation. The general pattern seems to be a consumer is subscribed to one or more queues, and Rabbit pushes to those queues based on rules that are highly configurable, but messages on an individual queue only get read completely "once" before being removed (this is a generalization, as there are a lot of options around confirmations and stuff).

This is a very simple image of a setup where each consumer only cares about distinct topics for them:

Whereas this is a more complex one, where a single topic gets split accross multiple queues. However, when we configure the queues, we need to know where to send the topics, and we still need one queue per consumer of that message

What I am envisioning instead is one huge message pipeline that all messages go through, with individual subscribers reading the messages relevant to them. Something closer to this concept:

My goal for this is to keep everything as loosely coupled as possible. With designs 1 or 2, we have to update the queueing system or exchange every time a new consumer gets added. With the third design, the queueing system is agnostic to what is consuming it, and instead just knows it needs to send messages. The consumers can then just read off the queue, grabbing copies of relevant messages to them.

Does this design make sense? We can't be the first ones to desire a system like this, as it should be a pretty common need (changes occur in a system, and multiple consumers need to act on them).

Answer 1

In the third diagram you are proposing, if Consumer 1 gets to the message first (and ACKS it), Consumers 2 and 3 will never receive it. If that is what you want then fine. If you want each Consumer to receive the message, then you need separate queue for each consumer.

The 3rd diagram is pretty much the scenario when you have multiple servers with the same consumer deployed. If Consumer 1 is deployed on 3 servers you don't care which servers gets to the message first.

To answer your concern, in diagram 1 and 2, your queuing system doesn't care about what queues are created. Or rather, your producer doesn't care because it's the consumer who should be creating and binding queues not the producer. The producer pushes a message with a routing key to an exchange. That's where its job ends. Then when you add a new consumer, it should create the queue and bind to the exchange with the routing key.

Answer 2

Perhaps, you can use something like Apache NIFI for message routing/transformation and kafka as a message broker. Both open source options.

1. Producers publish message to a Topic as per a normalized schema model > > >

2. Apache NIFI collects the message, does any transformation as per consumer needs from common schema model to consumer model (if desired) and routes it to specific topic for consumer > > >

3. Consumers (many-many) can be subscribed to different topics and consume those messages.

A few highlights, Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it. Queues by nature are not multi-subscriber. Topics on Kafka, also provide ordering and at least once delivered guarantees to consumers. https://kafka.apache.org/intro#kafka_mq

Answer 3

Typically the producer sticks the message onto a single queue that all consumers read from, with locking to ensure only one gets the message.

However, the 3rd scenario works brilliantly if all consumers receive the same message and decide whether they will process it or not. I've used that approach many times with great success, but you must have a way of differentiating which consumer will process the message and if you cannot, then all of them will. This may not be such a problem, for example a UI can use such a system and a message will be passed to all consumers (eg UI elements) that pass the criteria (eg they are under the mouse cursor when the event is raised)

So ultimately the answer here is "it depends" on what your messages are and how your consumers process them.

For further reading I'd look at the zeromq guide, they go into some detail about lots of different message queuing systems and the benefits/disadvantages of them.