I have a job that runs every hour. There can be
n identical Node.js microservices running in a stack. The job must be executed only once per hour, though, so I've implemented the following locking mechanism:
- once the job fires, SET a key with a random value in Redis, with params NX (if not exists) and PX (expiry time) of 30 sec
- if answer is OK then the lock is acquired, so the current service executes the job
- if the key has already been set or an error occurred while trying to set the key, then the answer will not be OK, thus re-try to acquire lock 4 more times, with 1 sec delay each time
- if lock hasn't been acquired after the re-tries, then exit w/o executing the job
- the service that has acquired the lock releases it after the job is finished
The problem is that when there are > 2 instances of the service, sometimes two services at a time acquire the lock (already on the first attempt, before the re-tries), so the mechanism isn't robust enough. I was thinking of a few solutions:
- set a random delay, eg. between 1000 and 3000 ms before the original attempt for acquiring lock. But this seems kinda' hackish...
- have an identity key for each service instance in Redis, thus allowing me to specify random delay according to the instance (sort of having a master instance). But I will also have to implement a flushing/cleaning of these keys, for when new instances are deployed to replace old ones, and that's too much.
- have an additional, single key in Redis which will hold an integer value to indicate how much of a delay a new instance should wait, and this value will be incremented each time after a lock is acquired. But there's a risk of race condition here as well...
I'm looking for suggestions on how to go about this issue. I'm not looking for external libraries, because the job is too simple to be needing such support.