Example:
A certain test is run five times on the current build, to compare it to release 1.0, where the same test was also run five times.
Build 1.4
22 ms, 26 ms, 23 ms, 25 ms, 20 ms
Release 1.0
15 ms, 18 ms, 16 ms, 20 ms, 17 ms
Question
The requirement is that Build 1.4 is no worse than Release 1.0.
How would I test that? I've seen different methods used, including simple average comparison and statistical T-Tests (assuming normal distributions).
None of the tools, i.e. testing frameworks and CI systems, I've come across provide anything to help with calculating and passing/failing such tests. Why? Performance testing seems to be popular, so how are people approving or rejecting them?
Often it's way too impractical or even impossible to go in and manually determine acceptable ranges or distributions for every test.