I have seen a reliability of the system expressed like this:

Rel=No of tests executed / Sum of crashes or other critical events

Does it make sense and what actually is the output?

  • Seriously? Your system is crashing, or emitting critical events? Where did you read it? – BЈовић Nov 4 '14 at 12:58
  • @BЈовић Quality metrics book (German). It is about critical systems and by that they mean "crashes or any other critical events with the same impact". – user144171 Nov 4 '14 at 13:04
  • 2
    Do you have a link to the book? – BЈовић Nov 4 '14 at 13:54

No. That metric displays a fundamental misunderstanding of both testing and reliability.

Testing can only ever prove the presence of bugs, but never the absence. A test suite demonstrates that a system is capable of functioning as expected (incl. known failure modes), but except in the most simplest cases can never prove that it will always work as designed. The size of a test suite is absolutely irrelevant. Larger test suites are not necessarily better, they could simply be a symptom of high system complexity. A small test suite could just mean that nobody bothered to write many tests.

A test suite with good coverage does translate into better reliability, because problems are found (and can be fixed) before the system is deployed. However, this requires good coverage (not only a large number of tests), and that somebody is actually fixing the problems. It is entirely possible to write a very large test suite with very little coverage, e.g. when only testing one specific subsystem, but leaving other subsystems untested.

Reliability is usually defined as the probability or frequency of failures, when using a system in a specified environment. If over a period of time T we observe n failures, then our reliability expressed as a frequency is roughly r = n/T. The number of tests is not a parameter here. However, it is possible to test reliability by replicating the specified environment and running the system there. After multiple runs, it can be decided with a certain confidence whether the system meets the reliability requirement. Unfortunately, this strategy is nearly worthless for testing deterministic programs, because programs don't have wear and tear.

Examples why the formula reliability = test suite size / number of failures won't work:

  • Let the systems A and B be identical, with the same test suite size (= 42). We expect the same reliability. We now let A run for two days, and observe three failures. We end up with a reliability of 42/3 = 14. Then we run B for two minutes, and observe one failure. We end up with a reliability of 42/1 = 42.

    Despite A and B being identical, we end up with different reliability metrics because the formula is not time-dependent.

  • Let system A be a simple system with a single test, and B a complex system with 1000 tests. We run both for the same amount of time, and observe one failure each. Therefore, A has the reliability index 1, whereas B has the reliability index 1000. Is B 1000 times more reliable than A? That's unlikely, considering that B probably has many more undetected failure modes.

| improve this answer | |
  • THanks! But the other way around (Failures/Time or No of Tests) is often mentioned in many books. – user144171 Nov 4 '14 at 13:37
  • @user144171 Well, the usual definition of reliability doesn't work when applied to short-running programs. Instead we can substitute other metrics of failure density, e.g. failures per invocation, which again makes little sense for deterministic programs. The metric failures/test cases describes how effective and reliable your test suite is, but does not disclose useful, comparable information of the system being tested. – amon Nov 4 '14 at 13:56
  • Yes, actually the Failures/test time or Failures/Test cases was a Failure rate calculation which at least can provide valuable information when comparing different releases of the same product. – user144171 Nov 4 '14 at 13:58
  • @user144171 I do not understand what kind of information provides you that value, since (to quote) "Testing can only ever prove the presence of bugs, but never the absence." – BЈовић Nov 4 '14 at 14:07
  • Your answer is fine, but IMHO your examples are flawed. If you can run the test suite B with 42 tests in 2 minutes, you will have to execute the test suite 24*60 times on system A to make this a two-days run, giving you a total of 60480 tests. Or did you mean something different? – Doc Brown Nov 4 '14 at 14:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.