0

I am new to RabbitMQ and herewith I want to make sure that I am not missing out on some advanced RabbitMQ feature or pattern I am not aware of.

I need to develop a reliable system that processes a few smallish RabbitMQ messages per second (~40/sec). My application receives those messages by subscribing to a RabbitMQ queue (which I do not control). The messages contain data as structured XML. The end goal here is to extract the data from the XML-messages and persist them into an SQL database for later use.

It is important for the queue operator that the messages are acknowledged as quickly as possible, before they are processed. This is more important than the processing latency, i.e. how long it takes for the messages to arrive in the SQL database.

Another requirement is that no message is lost under any circumstances. There is no way to get the information later on via a bulk request.

Currently I am thinking of writing a service which listens to the queue and writes every received message to an Azure Blob Storage with the messages binary body as blob content and the message headers as blob metadata. All fail-over strategies are in this service, too. That way I can record all messages and quickly acknowledge to the queue that I have received them. After that a second service can process them without any time constraints.

I am happy with such an architecture, but I am concerned that I am not aware of a RabbitMQ feature that can handle such a scenario in a more correct way or with less engineering effort.

Improve this question

1 Answer 1

Reset to default
1

This is not necessarily the best way to solve it either, but I thought Id throw the thought out there.

How about the initial consumer just consuming the message, immediately sending it to another queue (on the same rabbitmq) and then acking the message. That way you could then setup another set of consumers on the other queue that actually handles the messages.

The main improvement with this way of operating is that you dont introduce any additional dependencies or infrastructure. If you want to store it in azure blob storage you also have to handle that your calls to blob storage might fail, or they might be really slow. Whereas if you use the same rabbitmq then there is only a single thing that can fudge things up for you.

Improve this answer
4
  • The RabbitMQ queue from where my application receives the messages is controlled by an external system, not by me. What I am not sure on is whether I need to set up a separate RabbitMQ system on my infrastructure for your proposal to work. This would be an equally new dependency I'd have to manage.
    – Bruno Zell
    Apr 15, 2020 at 21:17
  • @BrunoZell Ah, that makes sense. But in that case, why involve blob storage instead of just saving the blob in sql server if sql server is the thing that you are gonna use anyway? You could just make a simple table without any indexes apart from PK and stick it in there. Shouldnt be noticeably much slower than blob storage and still allows you to minimize the number of moving parts. As a bonus, then you could also wrap a transaction around retrieving it, processing it and then deleting it from that table later on (which you cannot do with blobstorage).
    – wasatz
    Apr 16, 2020 at 5:40
  • Also possible, especially because the messages I receive are usually around 300B. In me specific case however it's nice to have a backup of all messages and in Blob storage it can sit for cheap. I take from this that there probably is no hidden RabbitMQ feature or pattern I am not aware of that can help me out here. Thanks :)
    – Bruno Zell
    Apr 16, 2020 at 12:52
  • Not any hidden pattern or feature that I know of, sorry.
    – wasatz
    Apr 19, 2020 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.