I have a web application that is currently of some different services, one instance each.
My current goal is to scale some of my services to ensure fail over. One big problem I foresee is that a service may execute the same task with all the instances, here's one example:
I have one service that sends a mail composed of different data grabbed from the internet. The mail is programmed by the user, either repetitive or one time. The planned mail is then put in database with the a datetime field stating when this should be done. My services are polling every x minutes to see if they have something to do, if they find something to be done they mark the job as handled and goes on doing it. The problem I face is between the time the service grabs the info from the database and the time the update is effective other instances may have grabbed the "false" information and send a second mail.
This could go the same way for another "observer" service which waits for incoming mails in an inbox. By polling the inbox and deleting the mail once handled, the same race condition could make 2 or more instances handle the received mail.
This question have one part of my problem and the proposed solutions would be good for me. But in this, it still makes the assumption that there is only one "watcher".
My other hypothesis is to have an elected master, or some communication channel between the different instances of a service to have only one of them send the message to the job queue...
I'd be happy to see how this kind of subject is usually tackled, surprisingly enough I couldn't find much information about that, or did not find the correct keywords to search for it.