2

I have a Bash script that processes files read from standard input, one pathname per line, and runs a CPU intensive task on each file. I happened to notice that even with four cores (grep -c ^processor /proc/cpuinfo) only about 25% of the processor was being utilized and that the disk IO rate was way lower than its capacity. Thus, there is scope to easily improve performance by splitting the input into four parts and giving each to a separate script.

The simplest approach that I can think of is to split the input into as many parts as the number of cores (obviously, I don't want the user to worry about the number of cores) using some variation of the split command and make changes to the script so that it can call itself passing a portion of the input.

What are some ideas in this regard?

  • Can you run the task in the background (as a daemon?) so that your script does not wait until a task finishes until you start the next task? Maybe this question is better suited for the unix/linux site? – Bent May 1 '16 at 16:08
  • Absolutely! I am thinking of doing something like presize & (say, name of the script is presize) from inside the script as many times as the number of cores. – pdp May 1 '16 at 16:12
  • 1
    Why not just start all of them? The kernel is probably better than the rest of us for schedulling/switching between tasks. If that turns out to be inefficient, start each task with a small delay in between, so that not too many tasks are running at the same time. But let the operating system do the work. – Bent May 1 '16 at 16:18
  • There is no doing the work of the kernel involved here. – pdp May 1 '16 at 16:43
1

A cool Bash feature is that a function can be called in a separate process. For example, if there is a function:

process_one_file() {
...

It can be called in a separate process (notice the ampersand in the end):

process_one_file file_1 &

I re-wrote the script to process 1-4 (as per user input) files in parallel. The main script would wait for each of the children to finish. When all the children get terminated, it would again launch the same number of parallel processes till there were no files to process. This design resulted in a 50% decrease in running time on my 4 core Intel i5 CPU based system.

|improve this answer|||||
0

If you can restructure your processing into multiple discrete steps with pipes connecting them (grep | sed | awk) that'll run each command in a separate process, giving you some parallelism for pretty much free.

Another option is GNU Parallel. It works vaguely like xargs but gives you parallel execution of the arguments. It'll pretty much 'magically' take a list of input files & fire off a process to handle each one, starting new processes as needed.

Both of these are fairly trivial ways to get some parallelism. The first option is better if you have a single large input, the second would be better if you have a bunch of different inputs.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.