Task Consumer/Processor architecture

Question

PROBLEM

We have various tasks in our system which can take up to 20 minutes. These tasks are generally started from the web interface and run on a new thread. This is obviously a terrible solution because the task could be recycled by IIS and it hogs up precious resources on the web server.

POSSIBLE SOLUTION

Hosting a TaskConsumer class in a windows service (or Azure Worker Role, in our case), which picks up tasks and processes them. This would handle the queuing and running of tasks.

New tasks are added to a database table (say Tasks) in the client code via some TaskManager class (TaskManager.Enqueue(ITaskable)), which are then picked up by the TaskConsumer (perhaps it is polling this table for additions?). This table would also be updated by the TaskConsumer while the task is executing (perhaps a state field - RUNNING, QUEUED, FAILED, COMPLETE).

To create a new type of task, one would implement an ITaskable interface. This would demand implementations for a Process() method, which would contains the logic to run that task.

A UI interface could show all active tasks and perhaps provide some management capabilities (e.g. cancel a task, restart a task, etc). This info can be retrieved from the TaskManager class (TaskManager.ActiveTasks), which essentially is reading from the Task database table.

The ITaskable concrete implementations would need to have serializable properties so to be stored in the database. This is neede because there would be specific parameters/data needed to run the task. For example, an ImportEmployees task would need to have the name of the file which was originally uploaded. This would need to be known at the time the consumer needs to execute the task (on desialization).

Is this a good approach to tackle this problem?

I am sure there must be many patterns out there for this kind of problem.

Answer 1

It is OK. You run into the problem that task consumers have to needlessly hammer the DB to look for tasks, task cancels, etc. I would have the task manager create an entry in the DB for the task so that the task consumer has a place to report results. Then use a Message queue like MSMQ or ActiveMQ to do the actual task consumer control. I probably wouldn't go for results reported over the message queue because you are going to have to write(and read) results to the DB just incase iis unloads your process. It much simpler to always read results from the DB.

If your DB load isn't that high or you don't want to complicate your deployment by throwing another bit of software in there, then your design will work just fine. Just be careful to not kill the DB having task consumers hammer it too hard.