1

I have a web application that is currently of some different services, one instance each.

My current goal is to scale some of my services to ensure fail over. One big problem I foresee is that a service may execute the same task with all the instances, here's one example:

I have one service that sends a mail composed of different data grabbed from the internet. The mail is programmed by the user, either repetitive or one time. The planned mail is then put in database with the a datetime field stating when this should be done. My services are polling every x minutes to see if they have something to do, if they find something to be done they mark the job as handled and goes on doing it. The problem I face is between the time the service grabs the info from the database and the time the update is effective other instances may have grabbed the "false" information and send a second mail.

This could go the same way for another "observer" service which waits for incoming mails in an inbox. By polling the inbox and deleting the mail once handled, the same race condition could make 2 or more instances handle the received mail.

This question have one part of my problem and the proposed solutions would be good for me. But in this, it still makes the assumption that there is only one "watcher".

My other hypothesis is to have an elected master, or some communication channel between the different instances of a service to have only one of them send the message to the job queue...

I'd be happy to see how this kind of subject is usually tackled, surprisingly enough I couldn't find much information about that, or did not find the correct keywords to search for it.

0

You should always try and program your applications so that you can run two instances at the same time with no problems.

In your example case its simple enough to prevent the dual sending of the email as you have a shared database which is used by all instances. Simply use a transaction to ensure that the same job can't be selected twice, updating a status column or something so that the second select doesn't pick up on the job.

In a more distributed scenario with message queues the problem is harder and potentially unsolvable. In order to prevent double processing you need some sort of shared lock, but the distributed architecture is specifically designed to avoid that kind of lock.

You can either have a single routing service, simple enough that it doesn't need to be horizontally scaled, which pulls jobs and assigns them to individual workers.

Or Segment the workers to avoid the possibility of duplication, say have one worker for email starting A, another for B etc

Or simply accept a low duplication possibility. After all most forms of queue or distributed DB will have some attempt at preventing it. You are probably only looking at edge cases where a duplicate message will be sent in practice.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.