I have a Bash script that processes files read from standard input, one pathname per line, and runs a CPU intensive task on each file. I happened to notice that even with four cores (
grep -c ^processor /proc/cpuinfo) only about 25% of the processor was being utilized and that the disk IO rate was way lower than its capacity. Thus, there is scope to easily improve performance by splitting the input into four parts and giving each to a separate script.
The simplest approach that I can think of is to split the input into as many parts as the number of cores (obviously, I don't want the user to worry about the number of cores) using some variation of the
split command and make changes to the script so that it can call itself passing a portion of the input.
What are some ideas in this regard?