0

I am trying to process sets of items (also, different sets have different items) in specific RabbitMQ consumers (one for each set of items) that would be created on-demand and are dispatched to the relevant instance (i.e., custom load-balancing depending on different heuristics).

To ensure that a set of items are processed exclusively by the same consumer, I know there is an exclusive flag. But putting that aside, I have no idea how to tackle this sort of architectural requirements.

I am wondering if there are already existing solutions for this kind of problem using RabbitMQ?

To some extent, I feel it all boils down to one single thing -- how the right service instance can say "hey I'm the one with enough load (or matching whatever heuristic) to be the right exclusive consumer"? I feel, I might need a system (i.e., each instance) that is capable of taking sneek peaks of all the queues available in RabbitMQ to decide what to process based on their own CPU. But that doesn't seem really possible.

Resources:

enter image description here

2

In the type of system that you're describing, the various workers need to be able to negotiate with each other and come to a consensus.

Here's one possible design that might meet your needs. I hope this makes sense.

In this design, all workers share an event-store used for task negotiation.


Job X is published to the fan-out exchange.
Worker-A and Worker-B both have the opportunity to claim the job.

Worker-A inserts an event similar to this:

  • Timestamp: 10:00:00
  • My CPU is at 45%.
  • I will take Job X if no one else with a CPU < 45% volunteers in the next 45 seconds.

(Worker-B hasn't seen the insert from Worker-A yet, but that doesn't matter.) Worker-B inserts an event similar to this:

  • Timestamp: 10:00:03
  • My CPU is at 50%
  • I will take Job X if no one else with a CPU < 50% volunteers in the next 45 seconds.

At 10:00:45, Worker-A reads the list of events.
Since no event was added to the ledger that would prevent Worker-A from taking the job, Worker-A starts working.

At 10:00:48, Worker-B reads the list of events.
Since there's a better candidate, Worker-B does not take the job.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.