1

Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, specifically multiple consumers subscribing to topics without destroying the message (i.e. multiple queues per topic).

The complications are as follows:

  1. We'd like the queue to be filled up by the producers even if all consumers are down
  2. We'd like to still be able to have multiple consumers be spun up to handle the work given to any one queue.

In my research, it's been bizzarely difficult to nail down an answer to these concerns.

  1. Some sources say that a topic exchange's queues and messages will remain persisted even when consumers are down as long as they've been started at least once, while others state that when the consumer stops consuming at any time the queue will stop receiving messages. I've been digging but cannot seem to find an authoritative answer.
  2. It appears as though we can hard-code the queue names via the consumer config, thus allowing multiple consumers to handle the work built up in a queue. Is this correct? Is it best practice?

If possible, please post an authoritative source for your answer so that others may find this in future reference. In my experience, RabbitMQ's docs are not the easiest to maneuver through, but I may just be doing it wrong.

Thank you.

8
  • I deleted my comments because I think I got the type of exchange wrong.
    – MetaFight
    Commented Apr 12, 2018 at 19:49
  • @MetaFight Copy that. I'll still look into the Fanout Exchange in more detail and test it out tomorrow. Up til now I've just discarded it as an option based on what I'd read related to our use case, but I shouldn't be so quick to do that Commented Apr 12, 2018 at 19:51
  • Are you looking for round-robin delivery of a topic's messages to a bunch of workers? If so, I believe that's just a durable queue bound to a direct exchange. If you have multiple clients connected to the same (non-exclusive) queue they'll receive the messages round-robin-style.
    – MetaFight
    Commented Apr 12, 2018 at 19:52
  • "when the consumer stops consuming at any time the queue will stop receiving messages" - Where did you read this? I have never heard of nor seen any such behaviour for RabbitMQ, as it is the queue's binding(s) which generally determine whether that queue is receiving messages or not. Indeed, if it were the case that a queue could no longer receive messages after its consumer had dropped, then that would make the whole concept of a queue rather useless, as one of their main advantages is the ability to store messages when no consumer is available. Commented Apr 12, 2018 at 21:24
  • @BenCottrell an apparently questionable YouTube video we came across: youtu.be/xWYEcOhremw?t=27s Commented Apr 13, 2018 at 12:52

2 Answers 2

2

What may be confusing is the RabbitMq pub/sub tutorial uses temporary anonymous queues created by the consumers : https://www.rabbitmq.com/tutorials/tutorial-three-spring-amqp.html, so yes, in this case, when the consumers are down, messages are NOT persisted waiting for consumers to go back up since the queues are not created when the consumers are not up.

But for your use case, you can use one fanout exchange and u can bind to this exchange any number of persistent named queues (maybe its already what you are doing), so that published messages are persisted even if consumers are down.

You create one queue per type of consuming (the type of action u want to do on the event/message, not the type of the message), but then u can add multiple consumers per queue dealing concurrently with the same type of action on the message for scaling up.

-2

Databases are best for persistence

It's a great option if you're already using a database. see https://todd-hubers.medium.com/you-really-can-replace-kafka-with-a-database-9e82a7c248a6

(Kafka is also a good option of course)

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.