Implementing distributed Semaphore for AWS workload

Question

We have a messaging gateway in AWS (Amazon Cloud) that comprises of a number of nodes (Linux servers) responsible for receiving, routing, despatching and escalating messages (emails, SMS' and voice calls). They all talk to a backend MySQL database on a dedicated machine.

Every few minutes we run a maintenance script - escalating stale messages, etc.

The problem is that we shouldn't run the maintenance on multiple nodes at the same time - hence I want to implement some sort of Distributed Semaphore that would be raised by the first node and if any other node attempts to run the script before the first one finishes it should simply exit. At the moment we run the script only on one node and have the cron job disabled on the others, but we prefer to have all the nodes with the same configuration.

On a single host I would probably use a lock file but how do I do that across multiple hosts? They don't have a shared filesystem and pretty much the only thing they share is the MySQL database. Also - if the process or node dies halfway through the processing the semaphore should eventually release or time out.

I prefer to use the database as the Semaphore holder but am also quite happy to use any other AWS service.

FWIW The script is written in Python. Not that I think it matters...

Thanks for any ideas!

Answer 1

If you want that only one machine at a time do something you can have a token and only the machine that "have" this token can launch the process. When a machine finish the process pass the token to other machine (randomly, round robin, to the machine who have less load...).

you can pass this token via AWS or store in the database. One problem to this approach its that the machine who holds the token crash and never pass the token to others. Perhaps you need something to recover this errors, distributed system are very complex things.

And precisely for this complexity, are you sure yo need something like that?, you can have one machine with the rol "maintenance_scripts_runner" and only this machine executes the script.