Hypothetically if every scenario were covered by an end-to-end tests, would unit tests still have any value?

Question

Note: I'm asking about the strategy behind unit / integration / end-to-end tests, not about classifying tests as one or the other.

More so in the past than present, it was expensive to write and run end-to-end tests for every possible scenario. Now though, for example with the increased use of test fixtures / emulators, or even lower latency / higher query limits to APIs, it's more feasible. But of course there's always gonna be human error. We can never be sure we thought of every scenario.

Still I'm asking anyway, hypothetically, given an oracle that tells us every possible scenario, or to put it another way, discounting scenarios other than exactly the ones with end-to-end tests, how might unit tests still be valuable?

What I'm wondering is if, unit-integration tests are kind of an "alternate approach" to thinking about tests, and that's the advantage of writing them even when aiming for "100%" end-to-end coverage: because they might catch a scenario missed by human error. But eliminate human error in thinking up scenarios, and what do they do?

Some ideas I can come up with (but I'm hoping for even stronger answers!)

• They encourage a useful coding methodology, e.g. TDD.
• On a breaking dependency (or dependent API) change, they reveal precisely where.

Answer 1

It's the second one.

Tests can demonstrate the capabilities of the system X as of now. But virtually always, requirements change and we will have to create a new System X' with subtle or big changes. Often, making these extensions breaks existing functionality, and then it is much more valuable to get the feedback "the email header input parser broke" than just "the Exxon import scenario has an error in it".

Answer 2

End-to-end tests and unit tests complete each other. They test code from different perspectives. End-to-end tests are focusing on a given feature, as it is implemented. Unit tests, on the other hand, test the actual algorithms in isolation. If you don't have unit tests:

• You cannot say with confidence that your algorithms are actually tested.

  Imagine an algorithm that miserably fails if either one of the input values is negative or zero, or the number of values is superior to 32,767 (because the algorithm is using a counter which is stored as a short which overflows at 2¹⁵). You may never find it, because every time this algorithm is called in your application, the caller ensures that the values are necessarily strictly positive. Additionally, it doesn't make sense from the business perspective to enter more than a hundred values. Your end-to-end tests are all green, but you still have a non-functional algorithm, which just happens to work in a given context.

  Why does it matter? Well, first at all, because of the possible regressions. Imagine that it starts making sense to enter thousands of values. Suddenly, you start receiving exceptions in production, because you never thought that this case would happen, and you never tested for it. If you were writing unit tests focusing on the algorithm, rather than the business rules, you may have thought about the edge case of more than 32,767 values.

  Moreover, if you take the algorithm as is, in order to reuse it somewhere else, you can't take the algorithm with its tests, because there aren't any. So you are grabbing a piece of code which is literally untested.

• You cannot pinpoint precisely the location of a bug.

  If an end-to-end test fails after you made a change, you are possibly clueless as to what exactly caused the failure. It shouldn't be that difficult if you did one very localized change, but could be quite challenging if you were merging a few hours of your work with a few hours of the work of your colleague (who by the way did a refactoring which spanned a few dozen of files).

  With a unit test, it's easy. It points you an exact location of your code which fails: a method, or maybe a series of methods.

• You don't have a quick feedback.

  When a change introduces a regression, it is valuable to have a nearly immediate feedback. If you know it in a matter of seconds, that's great, because you still remember what you were doing. If you learn a hour later that you broke something, you have to recall what you did a hour ago. If you learn it a month later (such as when tests are performed manually by a test department before the release), you possibly forgot every single detail of what you were doing, and you'll spend a lot of time just to recall what was the change about.

  Unit tests usually run within seconds. End-to-end tests are much slower. If you have both types of tests, you have the nearly immediate feedback from the unit tests, and a less immediate feedback from the other tests. If you don't have unit tests, you'll waste time every time there is a regression.

Answer 3

This question is about parts and whole:

• A system is made of many parts. Each part can be thoroughly tested with unit tests.
• The system as a whole can be tested with end-to-end tests, regardless of its inner structure.

Suppose you have a magic tool that finds all the possible scenarios to test. You then test every scenario and all pass: you’re in an ideal world where there is no need for unit test.

There is no difference between theory and practice... in theory.

In practice however, we all know that during its development, the system is quite often in a state where at least some end-to-end test would fail:

• 20 developers just changed 8 modules
• first, you need to wait that everyone finished his/her changes before being able to end-to-end test
• then, on the 1400 scenarios, 81 suddenly fail
• why ? and how to make it work again?

End-to-end tests won’t help you to know more about what’s going on: any of the 8 changed modules could have errors. Or all of them. Or they could all be correct, but their interaction produces another emergent behavior than the expected one.

Unit tests would allow you to quickly dive to the roots of your problem: the failed unit tests would quickly show you where it hurts. (in fact they would even do so before you can test the whole chain, making your quality efforts more time efficient). If all unit-tests passed and it still don’t work, it’s an emergent issue.

So in theory end-to-end test scenarios are sufficient if and only if your have identified all the scenarios and your system never fails.

If you’re not sure about both conditions, use unit tests together with end-to-end tests:

• the parts are easier to understand and easy to unit-test independently whenever needed;
• unit-tests are not subject to the same combinatorial explosion than end-to-end tests for the same coverage. If you have 3 components each with 4 cases, that’s 12 unit tests, but 16 end-to-end scenarios.
• some extreme conditions might only be verified in unit test because the conditions would never happen in real life, at least if all other parts would work well and prevent error propagation (i.e. not in a Titanic-like scenario of systemic failure)
• the whole results out of what emerges form the interaction between the parts. This is beyond unit-tests and this is why both kind of tests are required.

Answer 4

For me unit tests are as much about code design as they are about code quality. You suggest this in your first point and I don't think it's a trivial benefit.

Well designed code tends to be easier to unit test, so the activity of writing tests encourages the developer to write better code. It encourages a more conscious thought process about how the code should behave.

Unit tests are also a good way to document the expected behaviour of a function or method and it's often useful to read the test alongside the code.

So I would encourage unit testing for these reasons as well as pure code quality.

Answer 5

As of right now, end-to-end, or system tests are always orders of magnitudes slower, harder to implement and harder to run than unit tests.

The main advantage of unit tests is that they provide you confidence in your code in matter of seconds once you make a change in the code. They are also easier to run as they shouldn't require any changes to your system or access to external resources.

As a developer, I love using live testing tools (like nCrunch for .NET), which automatically runs the tests in the background as I type the code. And because the tests can be run fast and are trivially parallelized, it takes some 10-30 seconds to get test results once I stop typing. That is simply not possible with system or even integration tests.

So a solution with proper test pyramind of unit/integration/system tests still has significant feedback speed and ease of running over purely system test solution, assuming they provide same level of confidence in correctness of your software.