RabbitMQ: Topics and Persistence/Durability

Question

Right now I'm using Direct Exchanges (i.e. one queue per "topic", pretty much pushing messages directly to a queue). This works OK, but I'd like the flexibility of working with Topic Exchanges, specifically multiple consumers subscribing to topics without destroying the message (i.e. multiple queues per topic).

The complications are as follows:

1. We'd like the queue to be filled up by the producers even if all consumers are down
2. We'd like to still be able to have multiple consumers be spun up to handle the work given to any one queue.

In my research, it's been bizzarely difficult to nail down an answer to these concerns.

1. Some sources say that a topic exchange's queues and messages will remain persisted even when consumers are down as long as they've been started at least once, while others state that when the consumer stops consuming at any time the queue will stop receiving messages. I've been digging but cannot seem to find an authoritative answer.
2. It appears as though we can hard-code the queue names via the consumer config, thus allowing multiple consumers to handle the work built up in a queue. Is this correct? Is it best practice?

If possible, please post an authoritative source for your answer so that others may find this in future reference. In my experience, RabbitMQ's docs are not the easiest to maneuver through, but I may just be doing it wrong.

Thank you.

Answer 1

What may be confusing is the RabbitMq pub/sub tutorial uses temporary anonymous queues created by the consumers : https://www.rabbitmq.com/tutorials/tutorial-three-spring-amqp.html, so yes, in this case, when the consumers are down, messages are NOT persisted waiting for consumers to go back up since the queues are not created when the consumers are not up.

But for your use case, you can use one fanout exchange and u can bind to this exchange any number of persistent named queues (maybe its already what you are doing), so that published messages are persisted even if consumers are down.

You create one queue per type of consuming (the type of action u want to do on the event/message, not the type of the message), but then u can add multiple consumers per queue dealing concurrently with the same type of action on the message for scaling up.

Answer 2

Databases are best for persistence

It's a great option if you're already using a database. see https://todd-hubers.medium.com/you-really-can-replace-kafka-with-a-database-9e82a7c248a6

(Kafka is also a good option of course)