1

To clarify, as I'm not entirely sure how to summarise in the title.

I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc).

My redis cluster has a master and a couple of slaves. I insert a job onto the master and that is replicated onto the slaves.

I have 4 workers listening on these slaves.

The problem: I want a job to be only picked up once. But how this is currently setup is that one job will be replicated across the slaves which will then be picked up by multiple workers, duplicating the workload.

Would appreciate any suggestions on how to resolve this. I could use worker-specific queues and then schedule a job on a particular queue. But that will still replicate across slaves and build-up jobs that never get picked up which will need to be pruned.

I understand maybe redis isn't built for this kind of message passing and maybe something like rabbitmq would be a better fit. Open to any ideas.

  • Can you add a key with timeout like JOB_ID1<->WORKER1 so the workers first check to see if this job is already being worked upon by another worker? Timeout will help in case the worker crashes and another worker has to pick up this job.. – Srinivas Apr 17 '18 at 20:26
1

How are your workers picking up work? Are they polling redis or do they subscribe to a channel? You're right that this isn't something Redis is well suited for. The fact that it's durability guarantees aren't that great should also give you something to worry about if at least once delivery it's something you care about.

For reference, large scale message storage systems, like Kafka take the sharding approach. They occupy a different design space for sure, but you would not be going into uncharted territory by taking this route. You'd still have other problems though. Head of line blocking, uneven work distribution, issues when adding or removing clients etc. To top things off, you still need a mechanism to guard against duplicate work. That's a best practice in the sort of distributed systems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.