Design a system to send email at various times periodically? [closed]

Question

I am working on a feature that will allow the user to configure an email to be sent a time he chooses. Once he has saved this configuration, email will be delivered at his chosen time (or around that time) daily until he/she deactivates that configuration or deletes that. There can be many such configurations having different times.

So far I was doing something like this :

1. Run a cron job every 10 minutes.
2. Look for email configurations that are to be executed in (now - 10 minutes).
3. Send those emails and update status logs.

This used to work fine but has few pitfalls.

1. Emails configured at intersection times like 02:00 were not sent sometimes.

2. Due to a deployment or high CPU usage corn didn't run at all.

In both the above scenarios, email that are missed won't be picked again in next 24 hours.

How should I go forward to design a robust system that guarantees that all emails are sent.

If it matters I am using Django and a linux machine.

Answer 1

You can have a priority queue with a status flag (sent / unsent) for each job in the queue.

When you run the cron job, you will select from the priority queue as many unsent items as should be sent in the next 10 minutes.

When an e-mail is sent, you change the unsent status into sent.

You could also have some kind of counter for attempts, so if e-mail fails e.g. 10 times, you won't try anymore, but instead delete it from the queue.

The priority queue could be e.g. a table in a database having index on the send time, and then you select from it using the ORDER BY clause.

Answer 2

The standard solution for this is to store the "next process date/time" for each job. So when a user creates a job to run at 02:00, you schedule it for tomorrow at 02:00.

When the cron task runs (something like every 10-30 minutes), it just gets every* job that has a scheduled time in the past. Once a job has been processed, update the "next process time" for that job to tomorrow/next-week/whatever.

*As per the previous answer, it's generally advisable to decide how many emails a single run should send, and only process that many from the backlog. You do need to make sure you don't get into a situation where the backlog is never ending though...

Answer 3

A simple solution would just be to have the cron task or application write the time of the most recent successful execution to a file.

Then in place of now - 10 minutes it would between last success time and now(). Meaning you will never miss an email.

A more complex though thorough solution would be to store the last successful execution time on the schedule in the database and update that column each time you successfully send an email.

Then it just becomes a matter of querying for emails that have not been sent.