4

I am porting an event-driven architecture to a container environment (Docker) and I wonder how to take advantage of replication while using pub-sub: if I just increment replicas I end up with many process doing the same elaboration and I don't think is a good way to go. I thought to replace pub-sub with producer-consumer, so consumers waits for events from a queue and only one grabs it, but I would like to avoid tightly coupling.

Is there any solution to have a "load balanced" publish-subscribe? Also commercial tools can be checked.

Improve this question
3
  • Where are you seeing a tight coupling in the producer-consumer scenario?
    – Bart van Ingen Schenau
    Commented Sep 17, 2019 at 7:12
  • 1
    Have you had a look at kafka’s design ? It seems to address exactly this kind of issues. If scaling is limited, you can also consider splitting the processing part from the queue management so that you can multiply workers without adding the complexity of a distributed queue processing
    – Christophe
    Commented Sep 17, 2019 at 7:15
  • @BartvanIngenSchenau If N consumers exist (not replicas of the same but different logics), I need a queue for each of them, otherwise only one consumer handle the event. Am I wrong?
    – Marco Stramezzi
    Commented Sep 17, 2019 at 7:17

1 Answer 1

Reset to default
1

What you need is supported out of the box in RabbitMQ (and many other frameworks as well). This is one of the basic feature.

What are you needing here is a feature called Competing Consumers.

In RabbitMQ When you want one message to consumed by one of the many consumers. You set the bindings such that the message is routed from the exchange to one queue, and make multiple consumers listen to this queue. Then only one of the consumer will pick up the message. This is called competing consumers

When you want multiple consumers to consume one message. You set the bindings such that the message is routed to many queues from the exchange. Then the messages can be consumed by multiple consumer each listening to different queue

You could read https://www.rabbitmq.com/getstarted.html The scenarios are shown as 2 and 3 in the link above

Improve this answer
2
  • Thanks for your suggestion, but what you are showing is a producer-consumer implementation. I am asking if is possible to avoid it and keep using pub-sub.
    – Marco Stramezzi
    Commented Sep 24, 2019 at 6:24
  • @MarcoStramezzi, Yes... The link I showed was using RabbitMQ as a producer-consumer. RabbitMQ supports pub-sub patterns as well. You could check the link rabbitmq.com/tutorials/tutorial-three-python.html To get this to work we have to set the queue to exchange bindings, and we can have competing subscribers.
    – Nachiappan Kumarappan
    Commented Oct 4, 2019 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.