My system looks the following: enter image description here

I have N message producers, N RabbitMQ instances, N message consumers and 1 service. Producers produce the same messages while consumers consume the same messages. Here is the problem. Consumers must agree on who calls the service API, since calling the service API twice with the same message leads to an error. And I don't know how could consumers decide who calls the service.

I had some ideas. We could create a RabbitMQ cluster (with a plugin that detects duplicates) and unite all the consumers with the cluster. Consumers could send messages to the same queue using "direct" routing which will lead to "round-robin" message processing. But I'm not sure if there is a reliable and well-tested plugin that detects duplicated messages in RabbitMQ clusters.

Is there any other technique to solve this problem? I have heard about Apache Camel, but I can't say if it can save my day.

  • What problem? Is it not scaling in its current form? – Robert Harvey Nov 5 '19 at 20:00
  • Yes, I can see two comments made by you. I've edited my question. Please take a look. – Tony Nov 5 '19 at 20:05
  • If each consumer/producer line is independent, how does a different producer know the message has been duplicated? I'm not sure I quite understand why you're getting duplicate messages. – Robert Harvey Nov 5 '19 at 20:10
  • Producers are not independent. They all are connected to the Bitcoin p2p network. So they all have the same stream of messages(blocks). – Tony Nov 5 '19 at 20:17
  • Does having multiple producer/consumer queues provide any benefits? Or could you get away with just using one? You did say that they each has the same stream of messages anyway. – Robert Harvey Nov 5 '19 at 20:20

Your receiver should be idempotent and deal with duplication. Look at Idempotent Receiver Pattern. As you mentioned Apache Camel implement some features to deal with duplication.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.