Unit Testing order with contained & inherited objects

Question

I'm confused about unit testing order. Many posts in StackOverflow say that unit test should be independent and small. IMHO, in the case of aggregation or composition, the member objects should be tested before the container object.

Let's say I have a Test_Procedure class.
The Test_Procedure object contains one or more Test_Step class.
The Test_Step object contains Setup_Section, Actions_Section and Close_Section classes.

In the above scenario, in my understanding, the Setup_Section, Actions_Section, and Close_Section should be unit tested before the Test_Step class. And likewise, the Test_Step class should be unit tested before the Test_Procedure class. If there is a failure in the Setup_Section class, unit testing it first will show the direct cause. A unit testing of Test_Procedure class will fail, but it will show that the Test_Step class failed. One then has to find out why the Test_Step class failed, and finally, how the Test_Step step class failed. That is extra work.

IMHO, the Test_Step class should only be unit tested if it's member classes pass; otherwise it's a waste of time and effort.

The question also applies to inheritance; parent classes should be tested before child classes.

Google's Test Framework doesn't allow test ordering.

Why do people say that testing order does not matter for unit testing?

Answer 1

Because you are writing Module Tests.

There is a difference, and both have their advantages.

• A Unit test should be small, and constrained. Almost everything in a unit test is either Plain Old Data, or mocked out collaborators. The only "real" object is the one whose behaviour is being validated. This makes them fast, and when they fail, obvious that the unit's code itself is incorrect (assuming the data and the test itself are accurate).

• A Module test is about orchestration. It has several real objects (and perhaps a few mocked boundary objects) that are orchestrated to see if they can achieve a particular outcome. The presumption is that each individual object is "correct" but because they interact these interactions might not be "correct". As such when they fail, the clear reason for the failure is that the interactions are incorrect. This might require adjustments to the individual objects, or an alteration to the configuration of the orchestration (assuming the test setup and validation are correct).

Composition and Inheritance

Composition clearly puts the test into the Module category. There are several "real" objects working together to obtain an outcome.

Inheritance does too, though it is far from obvious when the test has only one "real" object. The reason is that the "real" object is a composition of the inherited with extra data and behaviour.

Nuances

Now you will need to squint a little. A class is by definition a Module, because it is a composition of data and behaviour. Yet tests written against the majority of classes that use just a single instance and mock out the collaborators will be considered unit tests.

The distinction depends on the level of analysis to some extent, testing a database engine even if presented via a single object cannot be considered a unit test.

Conversely a String of Characters (where both are classes) would probably fall into the Unit Test category, even though those Character objects would be separate "real" objects, and they would be orchestrated to work together.

I've no hard and fast rule to disambiguate where something stops being a unit test, and starts being a module test. My hunch centres around how much externally imposed orchestration occurs.

• A string has little to no externally imposed orchestration, it behaves the same ways regardless of its contents. Whenever a given character is searched for, regardless of whether it finds a location or not, it will always conduct the search in the same way.
• A database has much externally imposed orchestration, creating/dropping an index changes the speed of particular operations, which fields make keys alters the shape of storable data, even what tables exist. It has a meta-behaviour from which to produce a desired behaviour. Should the configuration be incorrect, the object will not behave correctly.

Ordering

Unit tests do not need a particular order of execution. Because they test for independent properties, and it is clear that their failure is because the object does not live up to its expectations.

Module tests do need an order of execution. They should not be run till after each of their "real" collaborators have been unit tested.

Test Value

However I would still execute the Module tests even if their Unit tests failed. At least as far as time permits.

Why?

Because there is the chance that the Module behaves as expected even if its components do not. This might illuminate the tests themselves as incorrect, or provide insight into why a particular configuration works around the component level defect.

Answer 2

Why do people say that testing order does not matter for unit testing?

Because in an ideal world, we would run all of the tests in parallel. This minimizes the amount of time it takes us to get feedback. For example, if we are doing test driven development, then we want to be keeping our attention on the design, and using tests to measure that design.

If you insist on introducing happens-before relationships between tests, then you increase the minimal latency.

There are some circumstances where ordering does make sense. For example, if a cheap test and an expensive test have significant overlap in coverage, you might want to ensure that the cheap test passes rather than "wasting" money on an expensive test that you know will fail.

Folks who are running "unit tests" as a part of their implementation activities aren't usually running expensive tests; the return on investment isn't there.

Answer 3

Test order shouldn't matter for unit testing, because you're meant to be testing distinct units. If your code is organized so that tests in the ParentTests class runs all the code tested by ChildTests, I would argue that either

• the "unit" in your code is the Parent class or
• the Parent and Child classes are too tightly coupled.

With classical, development-driven testing it can be very hard to end up with a clean architecture where responsibilities are sufficiently decoupled to avoid overlap between tests. TDD helps with this by forcing you to think about the architecture before starting implementation. Beware though: TDD does not give you the necessary understanding of refactoring to keep your architecture clean as it evolves, but it is possible.

Answer 4

There is no good reason to make unit tests depend on each other.

Why? Because you can store whatever the previous test would have provided.

Now it doesn't mater which test runs first. It only maters that you have a test that still produces exactly what you cached.

Squeeze time out of you unit tests. It's not your friend.