Code Coverage and Unit Tests nomenclature [closed]

Question

About tests:

I have the following view on nomenclature:

• Unit tests are the kind of testes where you have a function ExtractBacon, where there is a function with an entry parameter Pig and a return of type Bacon. What you test, is for the Pig with A, you expect a Bacon with feature X=1 and Y=2. And you assert that what you get when returning from that function is a Bacon with X=1, and Y=2.

• Code Coverage tests is the kind you guarantee that you pass the lines of the ExtractBacon. So if we have a function with statements like KillThePig, CutPigInMeatSlices and SmokeTheBaconMeatSlice, we want a proof the algorithm runs on all those statements.

Is my view correct?

Answer 1

Unit tests test a "unit" - generally a function/method, but maybe a class or even an entire module in some cases. But yes, the idea is that the test provides a known input (Pig A in your example) and verifies that it gets the expected output (Bacon with X=1,Y=2). Different test frameworks have different ways of verifying the outputs, but it's usually some kind of Assert statement. In most cases, several tests will all test the same method, with different parameters (or the tests themselves may be parameterized) - your example may have tests using Pig A, Pig B, and Pig C with different properties and thus different expected outputs. A whole different set of tests will test other methods in the class / module (maybe you have a method for MakeRibs() too)

Code coverage is a measure of how many / which lines of code get executed during your tests. It's usually expressed as a percentage of total lines of code, and may be broken down by module, class, and method. One major caveat with code coverage metrics is that it only measures how many lines of code are covered, not whether those lines of code are important. Refactoring can also cause confusion with code coverage results.

Consider in your example some pseudo-code:

public Bacon MakeBacon(Pig pigToSlaughter) {
    if (pigToSlaughter.isBabyPig) {
        return null;
    }

    KillThePig();
    CutPigInMeatSlices();
    SmokeBaconSlices();
}

If you only ever provide Pigs to your tests that are adult pigs, the tests will never execute that second line (returning null), leading to code coverage of 80% for this method. Provide a baby Pig in your test parameters and you now have 100% test coverage for this method. Depending on what the other methods in the class do, your test coverage for the class may still be dismal, though.

Now consider a slightly different version of your example code:

public Bacon MakeBacon(Pig pigToSlaughter) {
    if (pigToSlaughter.isBabyPig) {
        WashPig();
        DryPig();
        CurlPigsTailMore();
        PutBowOnPig();
        ReturnPigToSty();
        return new Bacon(weight=0);
    }

    KillThePig();
    CutPigInMeatSlices();
    SmokeBaconSlices();
}

Now, if your tests don't provide a baby Pig as a parameter, your code coverage for this method drops to 40% - because 6 out of 10 lines are never executed in tests. And the metrics for code coverage on your class are even worse.

Code coverage metrics can be useful as a tool in the build pipeline, but should be used with caution. It's very easy to write pointless tests solely to satisfy some benchmark (whether a minimum coverage that's part of a build step, or some metric-happy manager). On the flip side, it's also helpful to verify that vital methods / classes / modules are tested appropriately.

Answer 2

I commend you for using vivid and meaningful, if not kosher, names in your example. Pity X, Y, and A had to spoil it.

is my view correct?

Well when you say

Unit tests are the kind of testes where you have a function … And you assert that what you get when returning from that function is (what you expect)

Yes, that is the typical Arrange-Act-Assert pattern.

But, when you say

Code Coverage tests is the kind you guarantee …

Not quite. Code coverage is something you measure, not “test”. If someone says you have 80% code coverage it means some static analysis tool looked at the tests and the code and decided that the tests would execute 80% of the code. Meaning 20% of the code doesn’t get run by the tests. At best, that 20% compiles.

Achieving 100% code coverage is an efficient way to bankrupt a company. If that’s not your goal prioritize getting coverage over the interesting behavior code. Do not kill yourself covering dull obvious structure code. If you need an explanation of why look up the law of diminishing returns.

Answer 3

If you are looking for metaphors, imagine tests as industrial test benches.

A unit test takes a piece of code (sometimes a single function, others several more), isolates the piece from the rest of the code and stresses it. But, unlike in industrial engineering, we don't stress code applying pressure, torsion, voltages or rushing the application against a wall at 120KM/h. Instead, we formulate a hypothesis for this piece of code, and we prove it right.

given (piece) when (conditions) then (expectations)

We write a unit test for each possible condition, until we cover all of them (or a representative number) for this piece.

Integration tests are, somehow, the same but this time, there's not a single and isolated piece. This time, we test several pieces as a "whole". Imagine a test bench for car engines. Now, integrating "pieces" will generate new code, leading to new conditions and new hypotheses to be proven right.

As for the coverage, think about all the execution paths possible (conditions). By testing conditions, we have covered many of these paths (or the meaningful ones). Now bear this in mind; coverage alone doesn't prove anything.

When someone refers to a coverage test, is likely to be speaking about the need to test concrete conditions not covered yet, but it's not speaking about a specific type of test; it's just a way to speak.

Write a test "to cover up" functionalities X and Y.

If the above test is unitary, integration or E2E doesn't matter.