We have a messaging gateway in AWS (Amazon Cloud) that comprises of a number of nodes (Linux servers) responsible for receiving, routing, despatching and escalating messages (emails, SMS' and voice calls). They all talk to a backend MySQL database on a dedicated machine.
Every few minutes we run a maintenance script - escalating stale messages, etc.
The problem is that we shouldn't run the maintenance on multiple nodes at the same time - hence I want to implement some sort of Distributed Semaphore that would be raised by the first node and if any other node attempts to run the script before the first one finishes it should simply exit. At the moment we run the script only on one node and have the cron job disabled on the others, but we prefer to have all the nodes with the same configuration.
On a single host I would probably use a lock file but how do I do that across multiple hosts? They don't have a shared filesystem and pretty much the only thing they share is the MySQL database. Also - if the process or node dies halfway through the processing the semaphore should eventually release or time out.
I prefer to use the database as the Semaphore holder but am also quite happy to use any other AWS service.
FWIW The script is written in Python. Not that I think it matters...
Thanks for any ideas!