Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution which we used for this up to now was to have a mutex lock stored in database, that would prevent other instances from doing the job when lock is acquired.
In most cases it was enough for us, but the problem with databse locks is that we need to assume some timeout in case when the lock issuer dies before releasing it. This seems to be working well if the operation can be finished within the timeout, but in case we don't know how long the process will take, this causes the lock to be released too early.
I was thinking of adding some background job to each of the microservice instances, that would additionally monitor the locks acquired by the instance in-memory and execute the DB query to extend them every X seconds. Any problem occured while extending the lock (e.g. some edge case in which other instance was still able to acquire the lock) would make the long-running process be aborted.
Has anyone tried such approach? Was it successful?
