When scaling from a single server to elastic computing (many concurrent servers) how do timed tasks scale up?

Question

We have been in the process of scaling up my companies infrastructure, we have graduated from single servers and moved everything off to scalable AWS Ec2 servers that auto scale, there is one little item that has not scaled well though

Before, we had the server OS firing off a script that sent out emails via a cron task (ubuntu linux) unfortunately now because there are multiple concurrent servers, these cron tasks are firing concurrently, instead of sending 1 email they now send 2+ emails depending on how many servers are running at that given moment

Originally I tried creating something using one of our databases to check if the task had run in the last 5 minutes before allowing the task to run, but if they are both starting and hitting the database at the exact same time without latency then they will both run continue. This seems to be the case because multiple emails are still going out. I havent found a good solution yet, are there any ideas on what I can do to scale a task runner outside of moving this task to something serverless?

-- Edit:

sorry let me further explain what the app is doing (its not an email server) every week a group of people at my company is to receive a report of their progress, the application generates a report and saves it to the database, then emails the link to the user. Because these were being generated by a command line script using cron, at 6AM the computer runs the command line script, where it generates all 40+ reports and emails a link to each user. Because now there are multiple concurrent servers, each one runs the same script at the same time - generating an independent report, and sending the link to the user. From the users perspective they are receiving two emails with the same data.

Answer 1

Work sharing is a known problem. There are two primary challenges you have (with more challenges that depend on the solution you choose):

• Ensuring that processing only occurs once (i.e. only one of your instances processes a report at once)
• Ensuring the output of that processing only occurs once (i.e. only one email generated at the end)

A common solution is to have 1 process push work onto a queue, and multiple processes pull off the next work request and process it. If order is not important, this is an effective way of distributing load and ensuring things are only processed once. There are some error conditions you have to deal with:

• Use dead letter queuing to handle error conditions so you can reprocess reports that were missed
• This effort works for generating separate emails for each report

You can batch the emails by have another queue and a single process pulling off those report result messaging and consolidating the list. The emails would be generated on a schedule, and all the reports that exist for that email address would be included in the body of the message.

Answer 2

I think what i'm going to do is query AWS to determine how many duplicate servers are running concurrently, figure out how to numerate them and only allow the script to execute if the box is server #1

Answer 3

A somewhat canonical AWS answer would be to use CloudWatch events to schedule SQS messages. This combination handles most of the distributed system concerns for you. You can write familiar cron expressions to drop messages onto an SQS queue on a schedule. Your instances each attempt to pop messages off the queue on some cadence. The first instance to see the message will process it. The remainder of the servers will stay blissfully ignorant that there was an event in the queue. This all works regardless of the server/subscriber count.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html

I'm not familiar with EventBridge, mentioned in the docs. But, it just sounds like an alternative for CW events in this use case.