An application that I am working on and trying to optimize does data processing on large files, and so it alternates between being I/O bound with no CPU usage, and CPU bound with no I/O usage. It is block based processing, so this is same scenario happens across hundreds of iterations. The only strategy that I've come up with so far to mitigate this is to implement a producer/consumer scheme with one thread loading the data while another operates on it. Is there something else clever that I'm missing that I could be doing to make this more performant?
Producer/consumer might do it. In the "old days" it was called double-buffering, where the CPU cranks on the contents of one buffer while the I/O is filling the other.
What you need to do is make sure that all your available hardware is working as much as it can, keeping the CPU and the I/O busy simultaneously, as much as possible. It could be that the buffered I/O already does this for you.
Regardless, there is no point in having either the CPU or the I/O do anything unnecessary. So no matter what, I would do as much performance tuning as possible, cutting out the fat. The method I use is this.
ADDED: Just to show what I mean about BLAS, in particular the matrix-multiply routine DGEMM. It is usually assumed it is at or near the fastest possible speed. Maybe it is for a small number of multiplications of large matrices. In my case, I was doing a large number of multiplications of small matrices, like 4x4 or 5x5.
If I would take a small number, like five, of random-time stack samples, I would see something like this on three of them:
... in my code: CALL DGEMM ... in DGEMM: nota = lsame(transa,'N') ...
That means 60% of run time (very roughly) was spent checking the character flag
transa to see if the
A matrix needed to be transposed.
I knew that it did not, of course.
How to fix it?
Write my own matrix-multiply, in which small cases were hand-unrolled, and big cases would call DGEMM.
If the original running time was 100s, now it was about 40s.
That's a speedup ratio of 100/40 = 2.5
Then the same process could be repeated to find other such wastage.
Something else could have only been responsible for 20% of time or 20s, but after cutting out the original 60s, it is 20s out of 40s - easily spotted.
So fixing that gives an additional speedup ratio of 2.
See how it goes?
This is a typical use case for batch processing. You can use a framework like String Batch that even comes with predefined (simple) file readers and writers and you can configure the chunk size (nr. of lines to read).
If you are not using Java, you should still take a look at the documentation - maybe there exists a similar framework for your programming language, or you might want to implement it on your own.