Processing queued tasks in parallel?

Question

We have a system that publishes requests for processing into a database table. We also have a scheduled task that runs every minute and looks for pending requests in that table. If a request is found, the task processes it, marks it as complete, then picks up the next pending request, if any, and so on. If there are no more requests, the task terminates. A next task instance is started at the top of the next minute.

That all works fine, but we now have a requirement to process all of the requests as soon they are added to the database table, i.e., with a minimum delay. Which means we need to be able to process them in parallel, not one by one.

I am seeking ideas on how to implement this. One approach would be to have a monitoring thread in the task that would query the table for new requests periodically, and when a new request is found, span a processing thread for that request. Or, taking a similar approach, split the task in two executables - one for monitoring and another for processing. The former will start instances of the latter on demand.

Any thoughts?

Answer 1

I think you have the right idea - have a monitoring thread/process/application that does nothing but monitor. If it sees tasks that need processing, it hands them off as quickly as possible so it can get back to its job - monitoring.

A very long time ago a lot of Unix daemons (ftp servers, etc) were written that way. One thread would listen on the port for incoming messages. As soon as a message arrived, that thread would spawn a new thread to process the message and the original thread would go back to listening.

Since this is now the 21^st century, can you do this the easy way? Convert each task request into a web call then run that through a load balancer to a bunch of worker machines? It sounds counter-intuitive (since processing a single task might take a little longer) but you'll get some pretty easy to maintain parallelization.

Spawning threads on the local box is certainly also an option. If you're using Windows there are some tools that make it relatively easy to manage the thread pool for you so you don't have to keep track of how many threads you've spawned. If possible, try to create "fire & forget" (i.e., non locking) threads.

Answer 2

This has less to do with parallel then it has to do with database polling vs messaging.

You could make your current solution parallel by just adding a step where the process looks at the number of tasks and decides if it wants help. If it does it starts another task. Oh, you will want to mark a task as pending when something has started work but hasn't finished.

Bang, zowie, you're parallel. Big woop.

Everything I just said has to be done in a transactional way. That means it's got overhead. Sure it's parallel but that doesn't mean there isn't a faster way.

A messaging system would let clients start the processes that handle their requests directly and immediately. When resources are overwhelmed this would still queue. The problem is you have to rewrite the clients that used to talk to the database to now talk using the messaging system.

If rewriting the clients to talk to a messaging system is a no go, you can cheat. Put a packet sniffer on the database and parse out the requests. Soon as you spot them you can start work on them. Hurry because apparently 60 seconds is to long to wait.

Answer 3

Spawning new threads has considerable OS overhead o set up the environment, I would also look into having each task look for more work on the queue and only terminate when none found.