To clarify, as I'm not entirely sure how to summarise in the title.
I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc).
My redis cluster has a master and a couple of slaves. I insert a job onto the master and that is replicated onto the slaves.
I have 4 workers listening on these slaves.
The problem: I want a job to be only picked up once. But how this is currently setup is that one job will be replicated across the slaves which will then be picked up by multiple workers, duplicating the workload.
Would appreciate any suggestions on how to resolve this. I could use worker-specific queues and then schedule a job on a particular queue. But that will still replicate across slaves and build-up jobs that never get picked up which will need to be pruned.
I understand maybe redis isn't built for this kind of message passing and maybe something like rabbitmq would be a better fit. Open to any ideas.