Queue for tasks and notifications

Question

I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to bounce some of my ideas and get some feedback and trips.

So to set the stage, let's look at a use case:

We're creating a chat application that users can embed on their websites. When a user creates a new account in the system it should be running as a "trial" account for 30 days. Five days before the trial ends we should send an email reminder if the user hasn't configured payment. If the trial period ends and the user still has not configured payment, we should notify the user. When the service has been inactive for 2 days we want to check if the user is trying to use the chat-script on their website and notify them again if they do.

So the "batch approach" to this could maybe look something like this:

• Create a batch-job that fetches all users that have 5 days left of the trial. Check conditions, make sure we haven't already sent a reminder and send a reminder if needed.

• Create another batch -job that looks for users where the trial period has passed, change the status to "inactive" if payment has not been configured. Make sure we haven't already sent a reminder and send a reminder if needed.

• Create a batch-job that looks for users that have been inactive for two days, check if they are still using the script on the website, check that we haven't already notified the user and send notification if needed.

While this probably works fine I've been thinking about another approach to model these kinds of "future events" with the goal to make the process more explicit, easier to follow and easier to debug.

My current idea looks something like this:

When a new user is created we already know the timeline that we have for the events in the future. We could store some rows in a database to indicate the events we know that we might want to fire in the future. So, when the user is created we also queue these tasks:

• 25 days from now we want to send a reminder if payment has not been configured.
• 30 days from now we want to check the payment-status and inactivate the account if needed. At this point, if the account is inactivated we can schedule the follow-up task to check if the chat-script is still used on the users website two days later.

If the user at any point during the first 25 days configured payment we could either remove the upcoming "tasks" or set some status to ignore them. Same thing if the user decides to close the trial.

I can see some clear benefits of this:

• Each "task" represents just one thing that needs to be done. No need to handle problems that comes with batch-jobs like "what if it blows up in the middle of the processing?"

• We have a clear overview of "upcoming events" that we can query both on a wide "system level" and on the user-level.

• We could store task-execution result on the queue item and set it to "done", this way we have a log of each thing that happened.

• The concept is of cause reusable to many types of entities, not just users.

In terms of implementation, I'm still not sure how to implement this. I've been reading about distributed messaging systems like e.g. RabbitMQ or Azure Service Bus which obviously is great in terms of performance but I see a lot of value in the possibility to query the queue and sorting and also to keep a lot of events. A "poor man's solution" could be to pull the database for upcoming tasks and execute them and I guess that the sweet spot might be a combination where the details are stored in the DB and the queue is used for scheduled execution.

I would love to get some feedback here.

• Do you think that the "new approach" makes sense or would you argue against it? Why?
• How would you solve a similar use case?

Thank you so much for taking the time to read this and for any feedback!

Answer 1

The most problematic issue I see with your second approach is the implicit redundant encoding of certain business rules. The new account has a creation date and a payment status. The things which shall happen 25 days or 30 days after the creation can always be derived from that data. When you now create events for these rules beforehand, you produce extra effort and maybe issues in case the rules or conditions change in between.

For example, lets say the business people decide the 30 day period shall be extended to 35 days over christmas season, and they make this decision at the beginning of December. Or, users shall get the option of configuring if they really want to get a reminder after 25 days.

Sure, this can be handled all by updating the "future events", but the redundancy always yields a certain risk that this might lead to some inconsistent behaviour. Hence, I would usually prefer to store as few redundant data as possible, as long as this is not required to resolve performance issues. In my experience, that can help a lot to avoid error-prone situations.

Answer 2

My go-to pattern is to have a batch to poll the database at a relevant time and publish message to a queue or topic for eligible events. You can update the status of those items in the DB if it makes sense to do so.

The actual processing of events is queue or subscription driven.

This works well because you don't munge up timing, processing and updating statuses into a single, giant, failable task.

I think this approach is what you hinted at in your last option. It's a reasonable one to start with.

Answer 3

Scale is often a real issue, and the best design depends upon the scale needed. Countless projects have failed because the perfectly functional prototype is wholly inadequate once scaled up for production. The designers simply did not properly account for scale up front while they could still do something about the design.

The Unix "at" facility is excellent for scheduling work in the future. The job becomes independent, and will be run at the scheduled time, or afterwards if the system's not running at that particular time. However, each job is independent, and won't be able to share infrastructure like DB connections, etc. This might be too inefficient if there were a lot of these jobs. And you'd want to avoid creating a 'thundering herd', so some care might be needed there. (Scheduling exactly N days in the future, down to the second, might be sufficient to avoid this.) So, big plus for independence and using an already-there tool. A big minus (maybe) if high scale is needed.

And the point about business rules changing between initial scheduling and execution is well-made. Your DB can keep track of the jobs and cancel/reschedule them if there's a change, but that's additional complexity. Maybe at that time it becomes simpler for it to abandon "at" and do things more on its own?

My answer: It depends. :-(