I'm looking for a method to solve the following problem:

Assume there is a spool directory with a lot of files that need to be processed. There is a process A that is constantly writing new files to this folder and a process B that processes those files. So a classic producer / consumer pattern.

Unfortunately, at peak times, Process A writes new files faster than Process B is able to process them. So I would like to start multiple instances of process B to speed it up. However, these instances of process B would all work on the same spool directory and potentially get in the way.

Here is a pseudo code of process B:

while (running)
    take the first BATCH_SIZE files from spool
    process files

Is there a way to start multiple instances of process B without getting in the way? I've already thought about partitioning the amount of files, but I'd have to know the number of instances from the start. Let's say, I already have 1 instance running and I see that it is not enough ... Then I just want to boot up another instance temporarily without ending the first or somehow beforehand.

I hope that I have expressed myself understandable and I am very grateful for all ideas / helpful comments!

  • The easiest way is to provide an additional directory for Process B and put the overflow files there. – Robert Harvey Mar 20 '18 at 15:01
  • 8
    You can also create an intermediate consumer that grabs the files and passes them off to a set of worker processes. That way only one process is actually grabbing files. – Andrew Henle Mar 20 '18 at 15:03
  • 4
    Do you absolutely need a filesystem as synchronization between the process? Otherwise any queues implementation would resolve your problems, like RabbitMQ. You can change the number of consumers and RabbitMQ will take of the splitting of tasks and ordonnancing. – Patrick Mevzek Mar 20 '18 at 15:46
  • 2
    Do you have a way to identify which file is actively being processed by "B"? You could rename the file with an extension of .processing. That way another instance of "B" would ignore files being processed. – JeffO Mar 20 '18 at 15:58
  • 2
    Steve, you did forget to tell us how the processing state currently is managed. Process B shall not process the same file twice, even if it is just run sequently. Are processes files remove/renamed/logged/flagged? You got already two similar questions in the comments above, have been here in between but did not leave a word - it would be really helpful if you add a clarifying comment. – Doc Brown Mar 20 '18 at 18:05

As suggested above, using a tool like RabbitMQ that already does this sort of thing may make sense. But it also maybe overkill (learning curve of using the tool MAY exceed the value it brings to the problem).

A simple direct way to do what you want is to do advisory locking, or rename the file you are currently working on. If two processes try to rename (or lock) a file at the same time, one will will, and the other will lose. The one that wins continues and processes. The one that loses just picks another file.

To reduce the likelihood of collisions, you can add a little randomness to the selection of the next file to process, and you should get very few collisions.

  • 1
    You can reduce the likelihood of collisions to effectively zero by generating a GUID for the file name. – Robert Harvey Jul 17 '18 at 23:12
  • I think not. My understanding is that the input Q (spool directory) has a bunch of files in it, and a process that keeps creating them. Let's say they are all named something like XXXX.work (where XXXX is already a random number probably generated by something like mktemp()). My suggestion was to rename XXXX.work to XXXX.INPROGRESS, and have the code that looks for new tasks to read all directory entries that end in .WORK. Given that understanding of the original question and more detailed explanation of my solution, I don't think GUIDs would help or hurt. – Lewis Pringle Jul 18 '18 at 0:24
  • Since my comment about "few collisions" seemed to be misinterpretted, let me clarify that I meant that if you have N worker processes reading files in work Q, and using the same algorithm (say readdir) to find the next file to work on, they will invariably collide. If you instead have each process read uniform_int_random(1,N) files and take the last, they will much more rarely collide (they will miss some but you have to rewind directory and start at beginnning anyhow at end of Q so you will hit them then). – Lewis Pringle Jul 18 '18 at 0:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.