1

We have a messaging gateway in AWS (Amazon Cloud) that comprises of a number of nodes (Linux servers) responsible for receiving, routing, despatching and escalating messages (emails, SMS' and voice calls). They all talk to a backend MySQL database on a dedicated machine.

Every few minutes we run a maintenance script - escalating stale messages, etc.

The problem is that we shouldn't run the maintenance on multiple nodes at the same time - hence I want to implement some sort of Distributed Semaphore that would be raised by the first node and if any other node attempts to run the script before the first one finishes it should simply exit. At the moment we run the script only on one node and have the cron job disabled on the others, but we prefer to have all the nodes with the same configuration.

On a single host I would probably use a lock file but how do I do that across multiple hosts? They don't have a shared filesystem and pretty much the only thing they share is the MySQL database. Also - if the process or node dies halfway through the processing the semaphore should eventually release or time out.

I prefer to use the database as the Semaphore holder but am also quite happy to use any other AWS service.

FWIW The script is written in Python. Not that I think it matters...

Thanks for any ideas!

2
  • Use Amazon's Simple Queue Service!
    – TehShrike
    Commented Jul 28, 2014 at 3:33
  • @TehShrike keen to elaborate on high level how to use SQS for this purpose? Thanks!
    – MLu
    Commented Jul 28, 2014 at 5:10

1 Answer 1

1

If you want that only one machine at a time do something you can have a token and only the machine that "have" this token can launch the process. When a machine finish the process pass the token to other machine (randomly, round robin, to the machine who have less load...).

you can pass this token via AWS or store in the database. One problem to this approach its that the machine who holds the token crash and never pass the token to others. Perhaps you need something to recover this errors, distributed system are very complex things.

And precisely for this complexity, are you sure yo need something like that?, you can have one machine with the rol "maintenance_scripts_runner" and only this machine executes the script.

1
  • Thanks @AlfredoCasado that's what we do now - designate one node to the maintenance script runner. However to easy deal with AWS' autoscaling we would like to have all the nodes equal.
    – MLu
    Commented Jul 27, 2014 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.