How do I make sure a task run only once in a dockerized environment?

Question

I have a task that I need to make sure it only runs once in my dockerized environment (managed by k8s), running the post-upgrade script is one example.

Because each dockerized app can run the task and the docker containers may restart, I believe I need to store a state that indicating whether the task has run or not in a database, not just in memory. I also need to provide some locking mechanism to prevent multiple docker containers from running the task more than once.

The simple way is to use row lock to serialize access, only one dockerized app can read/write the record. For example just use sql SELECT ... FOR UPDATE, so the first docker container who gets the lock will check and run the task if necessary. All the other docker containers will just wait and get the set record later.

But I fell I can better than just serialize access. For example for the database supports SELECT ... FOR UPDATE NOWAIT I can use NOWAIT, so if a docker instance fails to get the lock that means one docker instance has got the lock and maybe have run the task, so it can just return, no need to wait for the lock then.

But not every database supports NOWAIT so I am thinking about how to implement a homemade solution to do that. The goal to make sure running the task only once.

Any suggestion ?

Answer 1

Assuming this is a "full" SQL database (i.e. with all the ACID guarantees), why not just use it as designed to get the lock?

UPDATE your_table
SET running_update = 1
WHERE running_update = 0

and then check the count of rows updated - the first run of the UPDATE statement (from any source) will actually update the row, while all subsequent statements will not make any change because running_update will already be 1. Every SQL database I'm aware of has the ability to get the count of rows affected by the previous UPDATE statement, although there's no standard for it - just read the fine manual for your database.

Once one task and one task only, the problem is trivial (ignoring the possibility of the update task failing) and you can just update some other data somewhere when it's completed.

Answer 2

A variant of Phil's answer: a three-state machine. NOT_YET_RUN -> RUNNING -> COMPLETE.

This is a little tricky to do properly, because you also need to record which instance is the one RUNNING, and have something check periodically to see if it has crashed and set it back to NOT_YET_RUN so that another runner can have a go.

Answer 3

You need a good way to implement a binary Semaphore or a Mutex. It's basically a safe way to say, "I've got this." There are lots of ways to do this. Perhaps the simplest is have a row in the table that gets changed when one of the instances takes the job. (pardon my pseudocode)

UPDATE JOBS SET OWNER="MY_ID" WHERE ID=$ AND OWNER=NULL

This update must be careful to only take the semaphore if the job is not already taken-- and database can easily serialize these requests. So check that you actually got the lock before proceeding:

SELECT OWNER FROM JOBS WHERE ID=$

The other important consideration is expiring the lock. Since docker containers can be ephemeral, consider the case where a container takes the job but then is shut down before actually doing this. It's probably easiest to incorporate a timestamp in the lock you are creating: (More pseudocode)

UPDATE JOBS 
SET OWNER="MY_ID", OWNED_AT=NOW() 
WHERE ID=$ AND OWNER=NULL OR OWNED_AT < (NOW()-1.minute)

Databases like Postgres actually have nice tools to address problems like this, but these basic examples should get you started.