0

We have been in the process of scaling up my companies infrastructure, we have graduated from single servers and moved everything off to scalable AWS Ec2 servers that auto scale, there is one little item that has not scaled well though

Before, we had the server OS firing off a script that sent out emails via a cron task (ubuntu linux) unfortunately now because there are multiple concurrent servers, these cron tasks are firing concurrently, instead of sending 1 email they now send 2+ emails depending on how many servers are running at that given moment

Originally I tried creating something using one of our databases to check if the task had run in the last 5 minutes before allowing the task to run, but if they are both starting and hitting the database at the exact same time without latency then they will both run continue. This seems to be the case because multiple emails are still going out. I havent found a good solution yet, are there any ideas on what I can do to scale a task runner outside of moving this task to something serverless?

-- Edit:

sorry let me further explain what the app is doing (its not an email server) every week a group of people at my company is to receive a report of their progress, the application generates a report and saves it to the database, then emails the link to the user. Because these were being generated by a command line script using cron, at 6AM the computer runs the command line script, where it generates all 40+ reports and emails a link to each user. Because now there are multiple concurrent servers, each one runs the same script at the same time - generating an independent report, and sending the link to the user. From the users perspective they are receiving two emails with the same data.

3
  • You are essentially asking how to make sure that each email is only sent by one server, correct? A simple way (no new software) is to store in the database: whether a server is busy with the email, and when that status was updated. A server can grab the email if no other server has marked it OR they marked it more than 15 minutes ago (indicating the other server crashed).
    – user253751
    Sep 13, 2021 at 8:30
  • It is impossible to remove all possibility of duplicate emails during a server crash (if the server that sent it crashed before it marked it as sent), but you can at least make it work properly in normal operation.
    – user253751
    Sep 13, 2021 at 8:32
  • 1
    By "timed task" I think you mean "scheduled task," i.e. something that is not time-sensitive. Is that correct? If so, why do you need to scale the server out if it's not real time?
    – John Wu
    Sep 13, 2021 at 20:52

3 Answers 3

0

I think what i'm going to do is query AWS to determine how many duplicate servers are running concurrently, figure out how to numerate them and only allow the script to execute if the box is server #1

3
  • 2
    That would not give you redundancy if server#1 crashes. Typically in this situation you would put the details of the work on a queue and have your workers read the queue and execute the job. The queuing mechanism ensures no duplicates and you get to use your multitude of servers giving speed and redundancy. Sep 14, 2021 at 5:08
  • i can absolutely slap them in a que - but two servers will be producing que'd items, once the item is que'd the worker would be working off the que, but there's no good way of them knowing if a second que'd item exists.... though I do agree the lack of redundancy is scary.
    – alilland
    Sep 14, 2021 at 18:39
  • Perhaps you can move that particular cron job to a separate machine that does not need to scale. Then modify the cron job so that the new machine executes the script remotely on a running machine.
    – John Douma
    Oct 14, 2021 at 15:51
0

Work sharing is a known problem. There are two primary challenges you have (with more challenges that depend on the solution you choose):

  • Ensuring that processing only occurs once (i.e. only one of your instances processes a report at once)
  • Ensuring the output of that processing only occurs once (i.e. only one email generated at the end)

A common solution is to have 1 process push work onto a queue, and multiple processes pull off the next work request and process it. If order is not important, this is an effective way of distributing load and ensuring things are only processed once. There are some error conditions you have to deal with:

  • Use dead letter queuing to handle error conditions so you can reprocess reports that were missed
  • This effort works for generating separate emails for each report

You can batch the emails by have another queue and a single process pulling off those report result messaging and consolidating the list. The emails would be generated on a schedule, and all the reports that exist for that email address would be included in the body of the message.

0

A somewhat canonical AWS answer would be to use CloudWatch events to schedule SQS messages. This combination handles most of the distributed system concerns for you. You can write familiar cron expressions to drop messages onto an SQS queue on a schedule. Your instances each attempt to pop messages off the queue on some cadence. The first instance to see the message will process it. The remainder of the servers will stay blissfully ignorant that there was an event in the queue. This all works regardless of the server/subscriber count.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html

I'm not familiar with EventBridge, mentioned in the docs. But, it just sounds like an alternative for CW events in this use case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.