1

I am writing a software that splits big files into smaller files and I have coded several solutions.

I am measuring the execution time of each of my solutions. (with threads, go routines, MPI etc) and want to objectively compare them.

If I run more then once the same solution, the execution time will be less and I understand that this happens because some of the data gets cached in the memory hierarchy (ram, or cpu registers etc).

I want to make the tests as more objective and reproducible as possible by removing these influences. I want to run each test with a clean slate.

If I restart the PC and measure performance again, then the ram is empty from the previous data and results are quite ok. I wonder if there is any way to do it without having to restore the PC?

What is the best way to make this kind of tests ? I want to do something like: Run a.exe and measure time, clean all ram, cpu register anything cached about this data Repeat N number of times test for a Do same thing for b,exe

Then I can calculate the average speed of a, the average speed of b and finally compare the data.

Please provide me some info as I am researching a lot and could not find any helpful resource. Optionally, I need programmable ways to achieve this. Some way to integrate in the benchmark pipeline the additional tools.

What I have tried so far:

  1. Restart PC just to make the point that caching was the issue
  2. Run the software inside a docker container, each time. Was good but very slow

Thanks in advance !

  • There are RAM optimizers which do little more free up memory not currently in use. You could probably get away with closing the program, running the RAM optimizer, then relaunching. – Neil Aug 30 '18 at 14:26
  • Thank you for the tip. Can you provide me one of these RAM optimizers that can potentially be integrated in a programmatic pipeline. I need to call that optionally by code in a loop any time I clean ram and run the consecutive execution. – Kristi Jorgji Aug 30 '18 at 14:29
  • 2
    We have solved this problem by running on a dedicated AWS box that can be spun up to run the tests. – enderland Aug 30 '18 at 17:02
  • BTW, a file size is in bytes (actually in gigabytes for large files), not in rows. – Basile Starynkevitch Aug 31 '18 at 9:03
  • I know, but I am splitten per line basis CSV and other text files for this particular project. Large text files are being split in smaller, big enough to be opened by editors. Bytes are checked, until a file is 50 mb regardless of number of lines then is emitted. Also updated question to remove number of bytes or lines as it is irrelevant to the question, so thanks for the insight. – Kristi Jorgji Aug 31 '18 at 9:15
3

If I run more then once the same solution, the execution time will be less and I understand that this happens because some of the data gets cached in the memory hierarchy (ram, or cpu registers etc).

So you should run several times (e.g. run five times exactly the same thing) the same "solution" and benchmark them all. The next question is what timing is the most relevant. You could choose the worst one (probably the first time), or you could consider an average of them, or ignore the worst and best runs and only care about the rest of them, etc...

In general, you cannot benchmark any software in exactly the same conditions because computer (and their operating systems) are not entirely deterministic, so you won't be able to reproduce exactly some running conditions. Hence, you need to make several benchmarks. Also "starting" or "cold-start" operation is not a typical running condition (but a special case), so you usually want to ignore it.

Remember that your hardware is non-deterministic: CPU cache behavior, CPU pipelining, superscalar processors with out-of-order execution, external interrupts -timers, networks, USB, disk, ...- and perhaps CPU frequency -limited when the chip is too hot- is changing without software control. Hence the kernel scheduler is behaving differently from one run to the next (because of preemptive scheduling, ...). Read also Operating Systems: three easy pieces for more about OSes. Some software layers (e.g. ASLR) could add more non-determinism.

In your case, I believe you want to consider the average time. In practice, it is very likely that some of the data is already "here" (e.g. in the page cache) when you would really use your program.

I dont think that measuring a "cold" state is realistic in your case. When you split a huge file, it is likely to have been generated (or downloaded, or obtained) a few seconds or minutes ago (why would you wait several hours before splitting it), so you really care more about a "warm" state, and in practice is it likely to be (partly) in your page cache.

Details are obviously computer, operating system, and file system specific. Don't expect your system to be deterministic and to give the same timings for several runs. So your benchmarks won't be exactly reproducible.

At last, your problem (splitting huge files of hundred of gigabytes each) is probably disk-IO-bound, not CPU bound, so the actual way of coding should not matter that much, at least if your buffers have suitable sizes (at least 128 kilobytes, and more likely a few megabytes; see setvbuf(3)...). If the files are not huge and could entirely fit in the page cache (e.g. if most files have a few gigabytes) things could be different.

BTW, on Linux, you might be interested by system calls like posix_fadvise(2) and/or readahead(2). When used properly, they could improve overall performance. And it seems that you are reinventing csplit(1) or split(1). Why are they not enough for your needs? Also, why do you need to optimize that much (remember that your developer's time costs more than the computer your program is running on).

Are you interested in splitting a thousand of files per day of a few gigabytes each, or splitting a dozen of files per day each of at least hundreds of gigabytes? These are two different problems! (I assume you have some ordinary desktop). And where do these files come from? How are they landing on your disk? What disk technology and file systems (SSD, rotating hard disks, remote filesystems) ?

PS. I am surprised by your question. Splitting a textual file (by e.g. thousands of lines) should not be an issue, and can be trivially coded in a satisfactory way (provided your buffers are large enough). I can't imagine a situation where such a splitting performance matters a lot (to the point of justifying several days of your work time). You need to explain your context, and motivate it much more. Of course byte size matters!

  • Sounds good. If I do not find any other solution, then I will increase number of execution times for each solution in order for the average to be more reliable. Or I take certain percentile, while removing 10% initial tests, and last 10% or something like that. As of now I use the average time of 10 tests (is very expensive and time consuming run one solution for example 1000 times in order to achieve the previous goal). Thank you! – Kristi Jorgji Aug 31 '18 at 9:01
  • 1
    128 kB is most definitely not a suitable buffer size. Try something around 16 MB. – gnasher729 Aug 31 '18 at 14:04
-1

My understanding / assumption is that you want to compare your 3 coding methods.

In order to compare them you want to be able to control the state of the machine so that it is consistent between runs or iterations.

In order to apply a scientific method to this, you need to run the experiment several times and make a grid of data results in order to analyse the data statistically.

The crucial part of this is running each test and making only one change each time.

Here's an example test script.

  1. Record PC state ( RAM, CPU temp, Disk type, cache etc. )
  2. Reboot PC
  3. Run program A
  4. Record time take to run, file size etc.
  5. Change 1 parameter and repeat

Changing 1 parameter might be switch to program B to compare types. Then changing to program C so that you can now plot a graph of A B C vs time. The only thing that has changed is the program type.

Then you can go about changing another parameter and running all the tests again.

You can then start designing your tests to support specific scenarios, e.g. Program A is faster for a file size over 10GB. Program B has a strategy for storing as much of the original file in RAM, so is the fastest for file size less than the amount of free RAM.

If you want to find out more search 'Scientific Method' - there is a Wiki page on it.

In terms of practical tools, there are automation tools out there, but I think that a simple batch file ( windows.bat ) will allow you to run your app, output the results to file and reboot.

  • Thank you for the ideas Mark. Like I state in the question, I am restarting the PC and I am wondering if it is a faster and more practical way to makes the test as more accurate as possible. – Kristi Jorgji Sep 24 '18 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.