I have a task that I need to make sure it only runs once in my dockerized environment (managed by k8s), running the post-upgrade script is one example.

Because each dockerized app can run the task and the docker containers may restart, I believe I need to store a state that indicating whether the task has run or not in a database, not just in memory. I also need to provide some locking mechanism to prevent multiple docker containers from running the task more than once.

The simple way is to use row lock to serialize access, only one dockerized app can read/write the record. For example just use sql SELECT ... FOR UPDATE, so the first docker container who gets the lock will check and run the task if necessary. All the other docker containers will just wait and get the set record later.

But I fell I can better than just serialize access. For example for the database supports SELECT ... FOR UPDATE NOWAIT I can use NOWAIT, so if a docker instance fails to get the lock that means one docker instance has got the lock and maybe have run the task, so it can just return, no need to wait for the lock then.

But not every database supports NOWAIT so I am thinking about how to implement a homemade solution to do that. The goal to make sure running the task only once.

Any suggestion ?

3 Answers 3


Assuming this is a "full" SQL database (i.e. with all the ACID guarantees), why not just use it as designed to get the lock?

UPDATE your_table
SET running_update = 1
WHERE running_update = 0

and then check the count of rows updated - the first run of the UPDATE statement (from any source) will actually update the row, while all subsequent statements will not make any change because running_update will already be 1. Every SQL database I'm aware of has the ability to get the count of rows affected by the previous UPDATE statement, although there's no standard for it - just read the fine manual for your database.

Once one task and one task only, the problem is trivial (ignoring the possibility of the update task failing) and you can just update some other data somewhere when it's completed.

  • I don't understand your answer. My goal is to make sure the task only run once, so I have to run the task once then set a record in database. I don't understand your algorithm, especially the words “check the count of rows updated” Commented Jun 1, 2021 at 7:48
  • 1
    This is how you get the lock to know which instance should run the upgrade task. Hopefully the problem is easy once you've done that. Commented Jun 1, 2021 at 8:22
  • With your updated answer now I understand, so you design 2 records in DB to do it! Commented Jun 1, 2021 at 8:37
  • @Qiulang: for guaranteeing the task is run only once, you don't have to set a flag in the after the task is completed. It is completely sufficient to the set the flag immediately when one process or container tries to start the task. Or do you want to try to run the task again if it does not complete in full the first time?
    – Doc Brown
    Commented Jun 1, 2021 at 8:40
  • @DocBrown right and that was actually one of reasons I didn't understand Phil's answer the first time. Commented Jun 1, 2021 at 8:42

A variant of Phil's answer: a three-state machine. NOT_YET_RUN -> RUNNING -> COMPLETE.

This is a little tricky to do properly, because you also need to record which instance is the one RUNNING, and have something check periodically to see if it has crashed and set it back to NOT_YET_RUN so that another runner can have a go.

  • Or set a timestamp when it starts running and assume it's failed if it takes more than (some time) to do the upgrade. Commented Jun 1, 2021 at 8:21
  • As I commented on him I don't understand his answer (yet). But I don't think I need to 3 states. My initial thought was if the check the record has not set, I will then run the task and set the record. If a docker container indeed crashes other containers can still do the job. Actually if I use "NOWAIT" other container just returns, then I may not be able to handle the crash situation. Commented Jun 1, 2021 at 8:24
  • The crash case is not my main concern because if a docker indeed crashes, k8s will restarts it then the dockerized app will run into the check & run part again. Commented Jun 1, 2021 at 8:29

You need a good way to implement a binary Semaphore or a Mutex. It's basically a safe way to say, "I've got this." There are lots of ways to do this. Perhaps the simplest is have a row in the table that gets changed when one of the instances takes the job. (pardon my pseudocode)


This update must be careful to only take the semaphore if the job is not already taken-- and database can easily serialize these requests. So check that you actually got the lock before proceeding:


The other important consideration is expiring the lock. Since docker containers can be ephemeral, consider the case where a container takes the job but then is shut down before actually doing this. It's probably easiest to incorporate a timestamp in the lock you are creating: (More pseudocode)


Databases like Postgres actually have nice tools to address problems like this, but these basic examples should get you started.

  • Thanks for answering my question. I don't think I need a counting semaphore but just a binary Semaphore or mutex. Phil's answer said "check the count of rows updated", the first docker updates the record gets 1 so it will run the task, other dockers get 0 so they don't run task. I don't see the difference between his answer and yours. Can you further elaborate it ? Commented Jun 2, 2021 at 6:28
  • Yes, it's just a mutex. I modified my answer. My answer (beyond Phils' good one) was mainly to give you a name ("mutex") to look up. (Sorry I whiffed that.) I also wanted to emphasize the timeout case. Perhaps I shouldn't have riffed on Phil's code as much-- what I'd really like to say is: dig into your database's manual and see what features they have for this. I know Postgres does this well.
    – ndp
    Commented Jun 2, 2021 at 6:35
  • Thanks! I actually asked a related question stackoverflow.com/questions/67770407/… that one is more specific about emulating "NOWAIT" while this one is more about running just once. Commented Jun 2, 2021 at 6:40
  • I further google nowait for Postgres and found it added nowait since 9.4 (2014), unlike mysql which only added it until 8.0. If I can use nowait I will just use nowait. Commented Jun 2, 2021 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.