An application that I am working on and trying to optimize does data processing on large files, and so it alternates between being I/O bound with no CPU usage, and CPU bound with no I/O usage. It is block based processing, so this is same scenario happens across hundreds of iterations. The only strategy that I've come up with so far to mitigate this is to implement a producer/consumer scheme with one thread loading the data while another operates on it. Is there something else clever that I'm missing that I could be doing to make this more performant?


Producer/consumer might do it. In the "old days" it was called double-buffering, where the CPU cranks on the contents of one buffer while the I/O is filling the other.

What you need to do is make sure that all your available hardware is working as much as it can, keeping the CPU and the I/O busy simultaneously, as much as possible. It could be that the buffered I/O already does this for you.

Regardless, there is no point in having either the CPU or the I/O do anything unnecessary. So no matter what, I would do as much performance tuning as possible, cutting out the fat. The method I use is this.

ADDED: Just to show what I mean about BLAS, in particular the matrix-multiply routine DGEMM. It is usually assumed it is at or near the fastest possible speed. Maybe it is for a small number of multiplications of large matrices. In my case, I was doing a large number of multiplications of small matrices, like 4x4 or 5x5.

If I would take a small number, like five, of random-time stack samples, I would see something like this on three of them:

in my code: CALL DGEMM ...
in DGEMM:   nota = lsame(transa,'N')

That means 60% of run time (very roughly) was spent checking the character flag transa to see if the A matrix needed to be transposed. I knew that it did not, of course. How to fix it? Write my own matrix-multiply, in which small cases were hand-unrolled, and big cases would call DGEMM. If the original running time was 100s, now it was about 40s. That's a speedup ratio of 100/40 = 2.5 Then the same process could be repeated to find other such wastage. Something else could have only been responsible for 20% of time or 20s, but after cutting out the original 60s, it is 20s out of 40s - easily spotted. So fixing that gives an additional speedup ratio of 2.

See how it goes?

  • 1
    Cool, I think we are on the same page. Having a dedicated thread reading in data into a circular buffer for another thread to operate on keeps the hardware busier. Unfortunately this is almost entirely matrix operations, so I'm already spending almost all of the process thread's time inside the blas library, not a lot of fat left to cut out. I'm also limited on both ends by an I/O library that isn't thread safe (HDF5). – alayers2 Aug 12 '16 at 20:52
  • @alayers2: The I/O could already be double-buffered, so layering in your own won't help. OTOH, if you're spending a lot of time in the BLAS library, there might be lots of room for speedup. For example, as I've found, it could well be that a large fraction of time goes into silly stuff, like checking the argument flags. The BLAS routines are written to save you the trouble of coding linear algebra. The price you pay for that convenience is they may be spending time checking lots of options that aren't relevant in your specific case. – Mike Dunlavey Aug 14 '16 at 22:02
  • That's a really interesting find with DGEMM. Generally speaking when it comes to the matrix ops, I try to keep the blocks as large as possible. About the smallest that I get is 1500 * 80000 (Since the input matrices to my program aren't always the same size, I dynamically size them to keep them about the same amount of memory). I would hope that I'm not incurring too much extra overhead using block sizes of ~1GB. – alayers2 Aug 16 '16 at 14:25
  • @alayers2: Reserving lots of space hits your RAM, so if your physical RAM is big enough, it should not cause page-thrashing. My points about DGEMM applied in the case of many multiplications of small matrices (regardless of reserved space size). That's just my experience. You could very well find a completely different way BLAS (or your code) is doing things you could cut out, because you're using it in a different way. I would be really surprised if, by sampling in an IDE or debugger, you can't find anything to slice out. – Mike Dunlavey Aug 16 '16 at 16:13

This is a typical use case for batch processing. You can use a framework like String Batch that even comes with predefined (simple) file readers and writers and you can configure the chunk size (nr. of lines to read).

If you are not using Java, you should still take a look at the documentation - maybe there exists a similar framework for your programming language, or you might want to implement it on your own.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.