I'm looking for a method to solve the following problem:
Assume there is a spool directory with a lot of files that need to be processed. There is a process A that is constantly writing new files to this folder and a process B that processes those files. So a classic producer / consumer pattern.
Unfortunately, at peak times, Process A writes new files faster than Process B is able to process them. So I would like to start multiple instances of process B to speed it up. However, these instances of process B would all work on the same spool directory and potentially get in the way.
Here is a pseudo code of process B:
while (running) take the first BATCH_SIZE files from spool process files sleep end
Is there a way to start multiple instances of process B without getting in the way? I've already thought about partitioning the amount of files, but I'd have to know the number of instances from the start. Let's say, I already have 1 instance running and I see that it is not enough ... Then I just want to boot up another instance temporarily without ending the first or somehow beforehand.
I hope that I have expressed myself understandable and I am very grateful for all ideas / helpful comments!