0

Consider unit testing for a program written in procedural style. Each unit test tests a single function.

Consider the following example, in which I use C_i(x) for a condition depending on the argument x and S_i for a statement.

def foo(x):
    if C_1(x): S_1
    if C_2(x): S_2
    if C_3(x): S_3
    if C_4(x): S_4

Assuming that all combinations of True and False values for the conditions can be achieved by passing different values of x, there are 2^4 possible execution paths and so 2^4 test cases are needed to achieve perfect path coverage.

Now, suppose we have decomposed this function:

def f_1(x):
    if C_1(x): S_1

def f_2(x):
    if C_2(x): S_2

def f_3(x):
    if C_3(x): S_3

def f_4(x):
    if C_4(x): S_4

def foo(x):
    f_1(x)
    f_2(x)
    f_3(x)
    f_4(x)

Now we have five units to test. For each of f_i, two test cases are needed. For foo, there is only one execution path. In all, we have 2*4 + 1 test cases.

So, is it true that decomposition can make it that perfect path coverage can be achieved with a polynomial number of test cases instead of an exponential number?

5
  • 1
    You are making the assumption that none of your f_<n> functions can possibly affect the result of any of the others. Nov 26, 2022 at 17:23
  • @PhilipKendall I understand what you mean. However, I am asking about the formal definition of the path coverage metric. Formally speaking, isn't there only one execution path in foo after decomposition (even if f_<n> are not independent)? Nov 26, 2022 at 18:04
  • 1
    It is important to understand that most of these metrics were defined before there were subroutines, modules, objects, classes, traits, interfaces, protocols, or any of this new-fangled nonsense. Just good old conditional GOTO. Nov 26, 2022 at 18:54
  • @JörgWMittag However, these metrics are used for unit testing and, to have unit testing, you need to have units in the first place. Nov 26, 2022 at 19:04
  • 1
    The answer is "You are correct". Not really a statement worth to be posted as a real answer.
    – Doc Brown
    Nov 26, 2022 at 21:08

1 Answer 1

3

Path coverage and code complexity measurements are metrics, and metrics can be gamed. Yes, you've correctly understood how a simple extract-function refactoring can drastically reduce the effort needed for achieving perfect coverage.

Some assorted thoughts related to this matter:

  • Assuming that the extracted functions represent meaningful actions in the problem domain and are truly independent, you may have actually improved this code.

  • I think testing private helper functions is a great approach, precisely because it can help keep the system under test smaller and more manageable. If you had to test the entire code only through the foo() function, then both variants of the code would require equivalent testing effort.

  • While path coverage can be a great metric as a white-box test design technique, such metrics aren't necessarily useful in practice. For a lot of projects even 80% line coverage is good, and aiming for 100% path coverage would be excessive.

    Coverage-guided test design can also lead to an overly mechanistic approach, resulting in tests that have low value. The purpose of tests isn't to achieve coverage, but to demonstrate that the software under test provides value to stakeholders, and secondarily to serve as a debugging tool. My current experience is that BDD-style test on the level of use cases have most value (“inverted test pyramid”). Coverage-guided approaches can be useful though for small fragments of very tricky code.

    For example, I once used such test design techniques to comprehensively test a number parsing routine in a banking software. To my horror (but luckily well before the software went live), this led me to tests that showed how some number formats weren't recognized correctly, potentially corrupting million-dollar currency values. But that was the 1% tricky code in the software, whereas the main business value was provided by other parts. Nowadays, I would leave such coverage-guided testing efforts to automated fuzzers, though.

2
  • "For a lot of projects even 80% line coverage is good". What would be an example of a software for public (rather than internal) consumption for which a buggy feature would not spell shame for the company that developed the software? Furthermore, if there are 20% of lines that are OK to go without testing, then maybe those lines should not have been written in the first place? Nov 28, 2022 at 8:30
  • "Nowadays, I would leave such coverage-guided testing efforts to automated fuzzers, though." Isn't it true that fuzz testing can only tell you whether the program crashed in response to an unexpected input, but not whether the output is correct if no crash occurred? Nov 28, 2022 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.