I need to send personalized emails to a filtered set of users (~100,000) on every fortnight. I am using Marketing cloud API for it. The way I am thinking of designing the system is as follows:

  • Scheduled job runs every fortnight and gets all user ids that match the filtering criteria
  • Then it queues the tasks to process individual user ids. Basically the task will generate the presonalized content and save it in marketing cloud using an API call
  • When all tasks are finished then it will again call marketing cloud API to send the email

I am going to use celery.chord to call these tasks one after the other

chord(add_user_data.s(user_id) for user_id in get_user_ids(limit=1000))(send_email)

I will control the number of tasks to add_user_data being queued because of limits on number of concurrent requests by marketing cloud API

Does the design seem okay? My main concerns are around what will happen if there is a worker restart because of a deployment

  • 3
    Your main concern should be how to avoid sending out duplicate e-mails after interruptions. Is that concern handled by the "marketing cloud API" or not, i.e. does it handle transactions or not?
    – tofro
    Commented Apr 4, 2022 at 9:52
  • so the database in marketing cloud where I am saving the personalized content handles duplication but the send email API doesn't. So if call the send email API twice in a row, it will trigger duplicate emails.
    – Prim
    Commented Apr 4, 2022 at 9:57

2 Answers 2


Doing an internet call 100,000 times in memory like this is asking for trouble.

Your program will crash or error and you wont know what's passed and what's failed.

You compound the problem when you add parallelism and email.

you need something like:

for each user
   //create email
for each email that is in status "new"
   //mark email sending
      //send email
      //mark email sent
      //mark email failed

Even that's not really enough, because a connection error while sending the smtp message might mean its sent or might mean its not. You want to be able to stop all the threads if something goes wrong.

If you want to send in parallel (and you probably do with 100,000), you'll need some sort of rate limiting as well, you don't want to try to open 100,000 network connections at the same time. but use a semaphore to keep the number to what your bandwidth can handle.

We can go further and talk about scheduled tasks, which ideally you should avoid. It's better to have a service constantly running and sending an email every few seconds over that two week period, than wait to the end and try to do it all in one go.

Obviously if you email is "Hot Off The Press Stock News!" you might have to, but for most things there is no real need to batch things up.

If you are using a "marketing cloud api" to send you emails, they should provide all the tools you need to manage user data, create templates, test emails, "campaigns" for groups of your users and handle all the actual sending for you, as well as reporting back on who read your emails, who many clicked the links, unsubscribe buttons etc etc

You shouldn't have to do much other than upload user data and click buttons.


You may want to consider decoupling user list collection, sending messages and processing errors.

I would put a queue in between for this. The scheduled job takes a user ids and writs them into the queue.

Many queues solutions allow listening the queue and process messages as they appear; you may have a listener tasks to actually call api per message.

If a specific message can't be delivered, e.g. due to an error - you can write that data into another queue for later analysis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.