Design periodic batch notification system

Question

Problem:

• I am trying to design a system which takes requests from customers in real time and log them in a database. For example - request to purchase an item.
• The customer then gets a unique request-ids for his request and also is asked to wait for 'X' days because the request is being processed.
• Then the system notifies the item providers by batching all these requests in one single email periodically (lets say 4 hours).
• Item providers update items in their repository and publish them to customers. And then customer gets a notification that item providers have responded for his request.

Design:

I'm thinking of using a workflow based system for this use case. With following components (high level) -

Synchronous API:
 - Receive request from customer. 
 - Record the request in the database.
 - Also start an asynchronous workflow, return request id as a response.

Workflow does the following :

Workflow Step 1:
 - Get the list of item providers
 - Notify them

Workflow Step 2: (This workflow steps gets executed after "X" days)
 - Send a response email back to the customer

But I'm having hard time to support the batching use-case. Is there any way I can efficiently batch the customer requests and send notification every 4 hours without having to write a cron job separately?

I'd appreciate any suggestions/help on this.

Answer 1

This is very simple: requests for process a task are made as normal - all that happens is an entry gets written to a DB with a timestamp on it. Trivial to implement.

Elsewhere in your program, you have a thread that goes to sleep for the required time, wakes up after 4 hours, reads all outstanding tasks and sends them off to suppliers.

The same applies for responses - the supplier sends a response into the system which writes it to a DB. Periodically a task wakes up, reads all outstanding responses and sends an email off to the customer.

Now you can optimise this, as you should see the steps are the same for both customer and supplier. You can set up one system that receives requests and writes data to the DB, and something else that is responsible for reading and sending responses. If you then optimise further, you can use an external system like cron to handle the wakeup processing - all it has to do is send a request to the system that triggers processing.

So you only have to code up a small amount of architecture that gets reused for the customer-request and supplier-response; and the wake-up and process entries. This is mainly the point of batch-processing, you keep it very simple and reuse the same process to process different data.

Answer 2

Your design seems that it can benefit from Asynchronous Messaging

Messages can be sent through different pipelines to multiple receiving applications and scheduled for execution at certain time (the scheduling can be a built-in feature of the message bus (middle-ware) or can be baked in your solution (reply with with a shceudled task id message and send the task off to a different application to be executed later)

Batching can happen within this architecture as a gateway app that collected messages with certain property values, group them together, and then send them as a different message (the batch message) every certain time frame.

Now I know that what I explained here is very abstract, but bear in mind also that explaining the whole architecture here wouldn't be practical either.

I hope that this answer could push you towards reading more about the architecture pattern for Messaging, and I'd be happy to answer other more specific questions later.

Answer 3

You can do that with - 1) Start the java/perl/shell script standalone program with cron at 00:01 and the program exits at 23:59 on every business day.

In the program you can read a configuration file to sleep for whatever interval you need. This makes it dynamic as you can change the config any time. Also you can add more functionality in the stand alone program in future.

So you are scheduling it only once. This gives you the flexibility of killing it any time at command prompt and restarting is easy.

I would advise against using database scheduler.