My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at this time (likely through Boomi or Idocs). We know next to nothing about RabbitMQ, except the general concepts of a message queue and what we've been able to determine from a high level look at their documentation. The general pattern seems to be a consumer is subscribed to one or more queues, and Rabbit pushes to those queues based on rules that are highly configurable, but messages on an individual queue only get read completely "once" before being removed (this is a generalization, as there are a lot of options around confirmations and stuff).

This is a very simple image of a setup where each consumer only cares about distinct topics for them: enter image description here

Whereas this is a more complex one, where a single topic gets split accross multiple queues. However, when we configure the queues, we need to know where to send the topics, and we still need one queue per consumer of that message

enter image description here

What I am envisioning instead is one huge message pipeline that all messages go through, with individual subscribers reading the messages relevant to them. Something closer to this concept:

enter image description here

My goal for this is to keep everything as loosely coupled as possible. With designs 1 or 2, we have to update the queueing system or exchange every time a new consumer gets added. With the third design, the queueing system is agnostic to what is consuming it, and instead just knows it needs to send messages. The consumers can then just read off the queue, grabbing copies of relevant messages to them.

Does this design make sense? We can't be the first ones to desire a system like this, as it should be a pretty common need (changes occur in a system, and multiple consumers need to act on them).

  • hmm no i think you will find that 'listen for a topic' means make a queue that is subscribed to that topic – Ewan Aug 17 '18 at 15:47
  • @Ewan wouldn't there need to be a queue then for each consumer of that topic? – Marshall Tigerus Aug 17 '18 at 18:31
  • no. what im saying is your consumers cant choose not to pull a message, or rather to do that they create a queue between them and the 'master queue' – Ewan Aug 17 '18 at 18:59
  • @Ewan that runs into the problem of the "master queue" having multiple readers for every message, which doesn't seem to be how it functions. – Marshall Tigerus Aug 17 '18 at 19:03
  • 1
    ..or rather, that the realisation of diagram 3 in rabbitMQ is diagram 2 – Ewan Aug 17 '18 at 19:10

In the third diagram you are proposing, if Consumer 1 gets to the message first (and ACKS it), Consumers 2 and 3 will never receive it. If that is what you want then fine. If you want each Consumer to receive the message, then you need separate queue for each consumer.

The 3rd diagram is pretty much the scenario when you have multiple servers with the same consumer deployed. If Consumer 1 is deployed on 3 servers you don't care which servers gets to the message first.

To answer your concern, in diagram 1 and 2, your queuing system doesn't care about what queues are created. Or rather, your producer doesn't care because it's the consumer who should be creating and binding queues not the producer. The producer pushes a message with a routing key to an exchange. That's where its job ends. Then when you add a new consumer, it should create the queue and bind to the exchange with the routing key.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.