How to architect master/slave store/messages and de-duplication?

Question

To clarify, as I'm not entirely sure how to summarise in the title.

I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc).

My redis cluster has a master and a couple of slaves. I insert a job onto the master and that is replicated onto the slaves.

I have 4 workers listening on these slaves.

The problem: I want a job to be only picked up once. But how this is currently setup is that one job will be replicated across the slaves which will then be picked up by multiple workers, duplicating the workload.

Would appreciate any suggestions on how to resolve this. I could use worker-specific queues and then schedule a job on a particular queue. But that will still replicate across slaves and build-up jobs that never get picked up which will need to be pruned.

I understand maybe redis isn't built for this kind of message passing and maybe something like rabbitmq would be a better fit. Open to any ideas.

Answer 1

How are your workers picking up work? Are they polling redis or do they subscribe to a channel? You're right that this isn't something Redis is well suited for. The fact that it's durability guarantees aren't that great should also give you something to worry about if at least once delivery it's something you care about.

For reference, large scale message storage systems, like Kafka take the sharding approach. They occupy a different design space for sure, but you would not be going into uncharted territory by taking this route. You'd still have other problems though. Head of line blocking, uneven work distribution, issues when adding or removing clients etc. To top things off, you still need a mechanism to guard against duplicate work. That's a best practice in the sort of distributed systems.