3

Example:

A certain test is run five times on the current build, to compare it to release 1.0, where the same test was also run five times.

Build 1.4

22 ms, 26 ms, 23 ms, 25 ms, 20 ms

Release 1.0

15 ms, 18 ms, 16 ms, 20 ms, 17 ms

Question

The requirement is that Build 1.4 is no worse than Release 1.0.

How would I test that? I've seen different methods used, including simple average comparison and statistical T-Tests (assuming normal distributions).

None of the tools, i.e. testing frameworks and CI systems, I've come across provide anything to help with calculating and passing/failing such tests. Why? Performance testing seems to be popular, so how are people approving or rejecting them?

Often it's way too impractical or even impossible to go in and manually determine acceptable ranges or distributions for every test.

Improve this question
4
  • 3
    If you don't know how fast is fast enough, why are you performing those performance tests? Are you also writing tests for behaviour that is unspecified?
    – Bart van Ingen Schenau
    May 14, 2015 at 15:13
  • Are your tests executing in an identical environment? Are you running tests concurrently (with themselves or other tests)?
    – enderland
    May 14, 2015 at 15:17
  • @BartvanIngenSchenau Sorry, clarified the requirement.
    – makhdumi
    May 14, 2015 at 15:21
  • @enderland Yes, but I don't think that's really relevant for my question. Noise is suppressed as much as possible.
    – makhdumi
    May 14, 2015 at 15:21

3 Answers 3

Reset to default
6

Performance testing seems to be popular

It's popular in theory. In practice, I've seen only a handful of automated performance tests done and they were done haphazardly.

How would I test that Build 1.4 is a pass/fail for this test?

You define a pass/fail criteria. If you need the tests to be consistent, define some failure criteria and measure. If you need the tests to be at least X fast (where X is ideally some metric gathered by usability testing) then do that. If not X worse than last build (not recommended in automated tests, since it requires memory of past builds) then do that.

It is trivial to write up helper code to deal with measurements and/or statistics. But you still need to define the criteria, just like any other test. It's like asking to automatically define pass/fail criteria for business rules - computers do a poor job at that since all they know about business rules is what you tell them.

Improve this answer
4
  • Right, sorry, the requirements are there. e.g. if it's considered a normal distribution, then Build X's values should be within some probability. It's trivial to calculate yes, but the infrastructure required - from keeping track of past build results to reporting, is rather large, and no CI systems I've seen support it. My question was why, but I guess your first point answers that.
    – makhdumi
    May 14, 2015 at 15:27
  • 1
    @Al-Muhandis This excellent answer makes a point that is important enough to be worth repeating and elaborating on a bit: you have to decide what the pass/fail criteria are. It really should be done when the rest of the project's requirements (the functional ones) are defined, and get a stamp of approval from the marketing teams and management as well.
    – skrrgwasme
    May 14, 2015 at 16:16
  • @skrrgwasme From the cases I've seen so far, the requirements have been that the build is no worse than the last release, or even sometimes, that it's "X times better" than the past release. I am wondering how people test this, i.e. data like in the example, when it's not built-in to any CI and/or testing frameworks. From this answer though it seems that it's manual or, when automatic, a hodgepodge of custom code and infrastructure (which concurs with what I've seen).
    – makhdumi
    May 14, 2015 at 17:27
  • @Al-Muhandis Ah. I think I misunderstood the point of your question. I thought you were asking more about defining pass criteria. But the point applies either way. It's up to you to decide how to test your systems, and it will vary between products and projects. Automatic is preferred, but if you're already working outside your test framework, it's probably not automated. If a performance benchmark is important enough to be specified, you should try to add support for it to your existing test infrastructure, whatever that may be. Manual testing may be necessary until that can be completed.
    – skrrgwasme
    May 14, 2015 at 17:39
3

The answer to your question is highly case-specific. You already seem to be aware of various performance testing techniques.

You must determine what your expected measurements are. It may be acceptable to start by saying "our expected measurement is the current measurement" and then just track changes over time. If it's "impractical or even impossible" to get an expected value then I don't see how the performance measurement would be meaningful.

Improve this answer
1
  • What I meant was to manually determine expected values. There are many tests, and with different configurations, the number of expected values increases exponentially.
    – makhdumi
    May 14, 2015 at 15:30
2

I can't promise that this is optimal, only that it's how I have been handling performance tests with moderate success.

Fairly substantial "real-world" integration tests get run under a timer, N repeats per test. Stash the results in a database. Periodically a set of results gets tagged as a waterline and subsequent runs are compared to that one, e.g. for a release, or when the testing hardware changes, or when we've implemented something cunning and everything runs faster.

The magic number used for comparison is the minimum of the results set. A fail occurs if it is greater than some fixed number (half a second or so) and is more than k% slower than the current waterline.

The minimum time seems to be pretty stable, but because the performance tests are run on a windows machine the worst case time jumps all over the shop. Periodically the test run collides with an antivirus scan or windows updates for instance. There's consequently an automated retry failed tests step which runs at the end to try to cut out some difficult to reproduce failures.

The manual tuning involved in the k% and the fixed time is unfortunate. I suspect a stronger solution involves looking for statistically significant trends and triggering a failure based on a probabilistic metric. Still, the above is simple yet still catches performance regressions.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.