Problem in generalization of benchmarking results
Firstly, if you run a benchmark program, any performance you observe by running that benchmark program is likely only applicable to your benchmark program.
Standardized benchmarking programs
To compare the performance of compiler outputs (in order to make a claim on the compiler's merit), you will likely need to run many different kinds of benchmark programs. To avoid any statistical sleight-of-hand, these benchmark programs shall come from some standardized source code, or they shall be well-known implementations of well-known algorithms.
You need to do your best to control every possible environmental influences on the program's execution time. In other words, any influences that aren't due to differences in the generated machine code emitted from your two versions of compilers, you must try to neutralize that.
An example of such interference is the CPU's own frequency variation. On modern Intel microprocessors, you must disable SpeedStep and Turbo Boost options. You should also change the power-saving settings to minimize fluctuations in CPU execution speed. The CPU itself should have more-than-enough thermal dissipation to prevent thermal throttling.
Memory and cache
You should familiarize yourself with the performance effects of memory access patterns and caching. You might be interested in reporting cold-cache results, warm-cache results, or both; you will need to apply appropriate preparation before and during benchmarks. This is a deep topic that will require you to devise a benchmarking methodology. This methodology need to be explained in every detail to your audience so that they can attempt to replicate and verify your results and claims.
OS, other processes, virtual memory
Minimize the number of other non-essential OS processes or applications during the benchmark.
A program's execution speed may be subject to the OS influence, which may be hard to control. Whatever attempts you make, document it in the benchmarking methodology as you report your results.
Modern OS typically use over-committing, which means that a program's request for more memory may be granted without actually making those physical memory space ready for use. Instead, the memory is only made ready when the program hits that address range for the first time. You may want to benchmark multiple scenarios with respect to virtual memory, just as you do with CPU cache effects.
Does execution speed depend on a program's input data?
It depends. In most cases, assume it does.
It is tempting to assume that a program that applies a simple operation to a huge array of primitive values would have an execution speed that does not depend on the array's content. Don't make this assumption.
Reporting confidence intervals
Report the confidence intervals as very low and very high percentiles. For example, 5% percentile and 95% percentile.
Do not assume the underlying measurement distribution to be Gaussian.
Understand that there will be outliers; but do not arbitrarily reject things as outliers - they may be rare events that do affect execution performance in general (meaning that users who run your benchmark programs will have a probability of seeing those rare events as well). This is why using confidence percentiles is important: it tells people exactly what fraction of outliers you are rejecting. Users who need stricter controls on outliers can request you to report stricter percentiles, say, 99.5% percentile, or even 99.99% percentile.
If you use Gaussian assumptions, such as summarizing results as mean and variance, you are destroying the part of statistical information needed to account for outliers.