1

For a test suite I have, each test has a failure rate (from 0 to 1), where 1 means it fails every time and 0 means it never fails, and a duration in ms (this is unbounded). I want a metric (I think either multiplication or division) that will capture the knowledge that tests with a low duration have a high value, and tests that have a high failure rate have a high value.

What is the appropriate metric for this?

Improve this question
3
  • 1
    What is the value of a test supposed to represent? I think the formula you'll need to use depends on how you're going to use it. – Tanner Swett Oct 21 '16 at 4:59
  • The time to run all tests is currently very long, and by finding low value tests we can get rid of them first. – Michael Hagar Oct 21 '16 at 15:50
  • Jerry Pournelle at one point, many years ago, talked about what he called the "figure of merit fallacy". The idea was that, if you combine all of your different evaluations into one single figure of merit, you risked making a really boneheaded decision, choosing an option that looked great on that one figure of merit but was actually lousy in the real world. – John R. Strohm Oct 21 '16 at 21:08
2

Aggregate your data in quartiles:

The first quartile (Q1) is defined as the middle number between the smallest number and the median of the data set. The second quartile (Q2) is the median of the data. The third quartile (Q3) is the middle value between the median and the highest value of the data set.

This means you will have something like:

| Runtime | FAILED     | PASSED    |
|---------|------------|-----------|
| 25%     | 20 seconds | 2 seconds |
| 50%     | 30 seconds | 3 seconds |
| 75%     | 33 seconds | 4 seconds |

This means that 25% of your failed tests are under 20 seconds. 50% are under 30 seconds, etc.

You can then directly compare them. Note that sometimes it might be useful for other thresholds, such as 90% and 95% or even 99% depending on the frequency of events.

Improve this answer
0

I think a good metric to use would be the "failure rate over time": if I were to run this test constantly (restarting the test immediately every time it finishes), how frequently would it fail?

To calculate this, just divide the probability of failure (the failure rate, as you've defined it) by the duration of the test. For example, if the probability of failure is 0.3, and the duration is 100 milliseconds, then the calculation is:

failure rate over time = (probability of failure) / (duration) = 0.3 / (100 milliseconds) = 0.003 per millisecond

Of course, this is a very small number. You could multiply this by 1,000 to get the failure rate per second, or you could multiply it by 3,600,000 to get the failure rate per hour.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.