I've really been struggling with this concept a lot in my head, and I was hoping I could get a poke in the right direction here...

I have a process that is supposed to take a certain number of phone numbers and send them all text messages (through Twilio). That's all fine and working...my problem is, I don't know how to ensure execution of each function, when there could be anywhere from 10 phone numbers to send to, to 10,000.

I don't think playing with the timeouts in my cloud environment would be best, because the time would be variable, so what if I set it too short? I just could never really know.

Or maybe timeouts is the best way to do this, and I'm just ignorant.

I guess my question boils down to:

How do I best execute a process that could take anywhere from 30 seconds to 6 hours, and make sure the entire process runs all the way through?


Nowadays, if I were designing something like this from scratch, I would use an Actor Model implementation like Akka. These systems provide the resiliency and monitoring that you require, out of the box.

To mark an operation as completed, just put a time stamp field in your database "messages" table that indicates as such. If a process goes down, you (or the Actor Model, in this case) can restart it and execute only the unfinished messages.

In the absence of Akka, you can use things like Watchdog Timers to monitor a process for failure.

Many cloud environments allow you to stand up Docker Containers containing the desired process. These containers can have a lifetime of a few seconds to indefinite. Check your cloud provider; surely there's some way to monitor these things.

In any case, periodically sending an "I am alive" message would give you the capability of restarting the container if it somehow crashes.

Further Reading
Introduction to Akka

| improve this answer | |

Break your input into fixed-size batches and process each batch independently. This gives you several benefits:

  • If one batch fails or times out, you only have to retry that batch without worrying about the others.
  • You can process these batches in parallel.
  • You can set a reasonable timeout for each batch since the size is bounded.

You didn't mention what kind of cloud environment you're using, but if you have access to a serverless platform such as AWS Lambda, you can simply put each phone number into a message queue and the platform will do the rest for you (batching, retries and parallel processing).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.