First, and maybe not so surprisingly, I found great performance differences when running the same code on two different machines (up to 6 x faster on one machine).
Then however and very much to my own surprise, running the exactly same code on the same machine on two different days showed great performance differences.
Performance measurement is a surprisingly hard topic and huge differences such as what you observed are very common. There is a lot of known measurement bias such as:
- OS noise, as suggested by DFord's answer
- OS policies (such as scheduler, memory mapper and IO-cache)
- Environment variables, even unused ones
- Memory alignment
- Memory localisation
Each of these parameters may greatly influence the performance of your program. If you want to prepare your program “for high performance” then:
Devise strategies to sanitize environment so that you gain some control over parameters 1-5 above.
Devise tests to determine which combination of parameters do well.
The key point here is that experiments need to be reproducible. If you are not able to reproduce an experiment, then it includes a source of randomness that you need to get a grasp on. And first of all, if you are not able to reproduce an experiment, then the results of the experiments are random and you probably do not want to take decisions based on random values, do you?
While this is not really the question you asked, I guess you are interested in performance measurement. Here are a few references I suggest to you:
http://compilers.iecc.com/comparch/article/96-02-165
Boehm commenting about execution speed for the same binary
varying by a factor of 2 depending on cache placement
http://www.inf.usi.ch/faculty/hauswirth/publications/asplos09.pdf
Producing Wrong Data Without Doing Anything Obviously Wrong!
It is very easy to read, contains surprising results, and give a very
good feeling of the difficulty of performance measurement and a few
methods to produce (more) reliable results.
http://plasma.cs.umass.edu/emery/stabilizer
“Stabilizer is a compiler and runtime system that enables statistically
rigorous performance evaluation.” There is a scientific publication
claiming to prove that -O3
and -O2
in gcc
are not distinguishable,
you will find plenty of interesting references in there.
As a final word, you just discovered with your measurements that wall clock does not provide a useful information when it comes to measure performance of programs, so that you will never get a useful answer to a question such as “how long runs my program?” and you have to consider more precise statements.