I have seen a reliability of the system expressed like this:
Rel=No of tests executed / Sum of crashes or other critical events
Does it make sense and what actually is the output?
Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.Sign up to join this community
No. That metric displays a fundamental misunderstanding of both testing and reliability.
Testing can only ever prove the presence of bugs, but never the absence. A test suite demonstrates that a system is capable of functioning as expected (incl. known failure modes), but except in the most simplest cases can never prove that it will always work as designed. The size of a test suite is absolutely irrelevant. Larger test suites are not necessarily better, they could simply be a symptom of high system complexity. A small test suite could just mean that nobody bothered to write many tests.
A test suite with good coverage does translate into better reliability, because problems are found (and can be fixed) before the system is deployed. However, this requires good coverage (not only a large number of tests), and that somebody is actually fixing the problems. It is entirely possible to write a very large test suite with very little coverage, e.g. when only testing one specific subsystem, but leaving other subsystems untested.
Reliability is usually defined as the probability or frequency of failures, when using a system in a specified environment. If over a period of time
T we observe
n failures, then our reliability expressed as a frequency is roughly
r = n/T. The number of tests is not a parameter here. However, it is possible to test reliability by replicating the specified environment and running the system there. After multiple runs, it can be decided with a certain confidence whether the system meets the reliability requirement. Unfortunately, this strategy is nearly worthless for testing deterministic programs, because programs don't have wear and tear.
Examples why the formula reliability = test suite size / number of failures won't work:
Let the systems A and B be identical, with the same test suite size (= 42). We expect the same reliability. We now let A run for two days, and observe three failures. We end up with a reliability of 42/3 = 14. Then we run B for two minutes, and observe one failure. We end up with a reliability of 42/1 = 42.
Despite A and B being identical, we end up with different reliability metrics because the formula is not time-dependent.
Let system A be a simple system with a single test, and B a complex system with 1000 tests. We run both for the same amount of time, and observe one failure each. Therefore, A has the reliability index 1, whereas B has the reliability index 1000. Is B 1000 times more reliable than A? That's unlikely, considering that B probably has many more undetected failure modes.