0

I hope that's a proper place to ask my question.

I am wondering how is best to integrate RabbitMq (or any other message broker) into my project.

There are 2 options:

  1. Simply collect all messages, whatever comes from my other services and dump all into Rabbit

Option1

  1. Collect messages into intermediate service, to ensure "contract", which will then forward them to rabbit in some particular form

Option2

While first option seems to be native, it may (?) create problems, if we try later to switch from Rabbit to smth else, like Kafka (or not?), the second seems to defeat Rabbits purpose.

I actually want to know, if the second option has a right to exist at all and why? Is it standart to have some extra service before message broker or not (and what could be the reason for it)?

7
  • Do you realise that we can not tell you "what you need" right? You already answered yourself tho. "yes" if that "coupling" is a problem for you. "No" if it's not. Anyways, I'm afraid of asking for opinions is offtopic.
    – Laiv
    Jun 30 at 13:26
  • Yeah, I just wanted some discussion, I do not know where else can I discuss this :) Sometimes, people ask questions like "what is better..." or "what is the difference.." here..I think my belongs to this category. I mean how else do you discuss/decide on architecture?
    – MStikh
    Jun 30 at 13:29
  • Maybe, you get some answers if you tell more about the current architecture, the number of existing services, the possible number of services in the future, the problems you are facing today and what you want to achieve with the next design, etc. You know, a bit more context and put the focus on those things that concern you.
    – Laiv
    Jun 30 at 13:33
  • 1
    I was actually hoping, that someone will come and say that it is really stupid to have message collecting service before message broker (option 2)..
    – MStikh
    Jun 30 at 13:38
  • 1
    There are no stupid solutions. Just adequate or inadequate ones. You are the only one who can say if that service is solving something NOW (not tomorrow). If we try later to switch from Rabbit to smth else, like Kafka is only guesswork then it's unlikely you need it. Even if it's possible, you are the only one in a position to say if the complexity introduced by the new service worth its implementation.
    – Laiv
    Jun 30 at 13:44
0

OK, so rabbit or MSMQ in general doesn't enforce type safety of messages you send to queues.

So if service1 is sending :

{
   name : ewan
}

and service2 is sending :

{
   customername: ewan
}

to the same queue, then whatever is reading those messages has got a problem to deal with.

With that in mind you can see why an intermediate service might seem reasonable.

However, How are you communicating with that intermediary? If its just an in memory class wrapping the MQ send; I think everyone does that. Its just normal coding.

If if a microservice you are calling over http, then you have lost some of the flexibility and asynchronousness of writing to a queue. Plus, are you really enforcing type safety any more stringently? Lots of the same things that would cause errors with just queues, mismatched deployed versions etc can still happen to you with queue + protecting api.

I would say its overkill. Have a library class that enforces the type -> queue name and use that, but go direct to the queue. Don't add intermediary transport protocols.

There is another option which i think is fairly normal and fits the paradime of queues, which is to have an intermediary queue worker route the messages

so you can do things like:

p1 -> q1
p2 -> q1

q1 -> w1
w1 message is type A -> qA
w1 message is type B -> qB
w1 message is unknown -> eError
w1 message is A but queue A is full? -> send alert -> qOverflow
w1 we are moving to kafka even though its a streaming db not a mq -> duplicate message to kafka

etc

3
  • Hi, Thank you for detailed answer. Just as comment: >If its just an in memory class wrapping the MQ send; That means that if something changes on RabbitMq end (it will be replaced or smth), this class will have to be overwritten in every service, and if there are many services?
    – MStikh
    Jul 1 at 8:17
  • "intermediary queue worker" is a cool idea. Is that something existing within RabbitMQ? Can you please give some link to that?
    – MStikh
    Jul 1 at 8:20
  • you just have to write a program that pulls from one queue, applies logic and posts to others
    – Ewan
    Jul 1 at 13:11
1

I'm going to extrapolate a little bit from your question and assume that the reason you even want a message queue architecture is that p1, p2, and p3 generate a lot of asynchronous requests and you don't want p4 to be a bottleneck.

In that case your first idea is far preferable. The idea is that a zillion asynchronous requests may pile up in the queueing service so p4 can simply do its job processing one request at a time, as fast as it can.

In the second scenario, you're forcing p4 to talk to three contending processes and somehow prioritize them and perhaps respond to them in a timely fashion. Which is the type of concern that RabbitMQ is supposed to relieve you of.

I think it's quite likely that your first idea is the better one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.