I am trying to process sets of items (also, different sets have different items) in specific RabbitMQ consumers (one for each set of items) that would be created on-demand and are dispatched to the relevant instance (i.e., custom load-balancing depending on different heuristics).
To ensure that a set of items are processed exclusively by the same consumer, I know there is an exclusive flag. But putting that aside, I have no idea how to tackle this sort of architectural requirements.
I am wondering if there are already existing solutions for this kind of problem using RabbitMQ?
To some extent, I feel it all boils down to one single thing -- how the right service instance can say "hey I'm the one with enough load (or matching whatever heuristic) to be the right exclusive consumer"? I feel, I might need a system (i.e., each instance) that is capable of taking sneek peaks of all the queues available in RabbitMQ to decide what to process based on their own CPU. But that doesn't seem really possible.