How to do achieve parallelism in a console application

Question

I have a .net web application, let's say A and a .net console application, let's say B.In A, I have a queue with multiple jobs to be done by B.Once I select the jobs from A, a queue gets created. Each job in the queue gets picked up by the console application B sequentially in FIFO and performed/processed. If I have let's say 10 jobs in the queue, B will pick up the first job, process it completely, then pick up the second job performs it, then picks up the third job and so on... The applications use an MSMQ (Microsoft Message Queue). Application A will add jobs to it and application B is continuously listening to the queue. If any job gets added to the queue by A, B will pick it up and executes it and pick up next if there is any and so on.

What I want is to achieve parallelism for these jobs from the console application. Can I use multithreading for that or should I create instances(let's say 5 instances) of the console application to execute the jobs in parallel?

Is there any better approach to achieve the same? please suggest any resources to start with.

Answer 1

First you need to decide if the jobs have no dependencies among each other. If they need to be executed in a certain order, executing them in parallel will not be possible, at least not in full.

But let us assume each of this jobs work on a completely different, isolated data set, so parallel execution should be possible, at least in theory.

So the next question you have to answer here is:

• shall the jobs be executed by multiple threads inside of B? Or does it make more sense to split up B into a worker process and a message dispatcher? So the dispatcher can listen at the queue and spin up the new instances of the worker.

The answer to this depends on some factors. The first thing I would look at is probably, is B currently written in a thread safe manner, or - at least - can it easily be made threadsafe?

If the answer is clearly no, then using multiple processes will make more sense. If the answer is yes, I would actually try to start with the most simple solution, which is to use multithreading, and switch only to multiple processes when multithreading turns out not to be sufficient. See also this former SE.SE question:

Performance: Is There a Reason to Use Processes over Threads?