1

I took out the request for a paper, thank you for spending so much time on this.

hopefully this edit is better.

I use the word "complexity" meaning "cyclomatic complexity" so if you want me to put that all in there i will. I am specifically looking for an academic paper also, and I think that's relevant to engineering, but however you need me to phrase it i'm happy to make changes.

Thanks, John


I am convinced (and have been trying to convince others) that the cyclomatic complexity of unit testing implies that unit testing has a comparatively low cost and that it prevents multiple executions of "running the system" as a method of testing things. By preventing this overuse of what I'll call "system testing", unit testing saves costs. That's my argument. I know unit testing doesn't replace system testing, but often it's argued that unit testing isn't needed because we have the system tests.

I started some of the (very approximate and rough) math for my argument, and the basic argument I would make is that unit testing cyclomatic complexity is, for instance (random choice of numbers): = f(x) g(y), C(f(x)) = 3, C(g(y)) = 3, so the complexity of unit testing is the sum, or 6, and the complexity of system testing for C(f(x) g(x)) is more like C(3*3) or 9.

Therefore, system testing is O(n^Z) where n is the complexity and Z is the number of functions (obviously realistic systems are larger but this is a base case)

And unit testing is O(n)

I dont know if this is right, but it's my intuition and I think that's a small enough problem to work out, hence I'd love to see a proof of anything even slightly related to this, in some sort of formal ... format? that seems like a bad sentence, but you get it.

This would end the argument in my head, and at the very least I could then argue that continuous O(n) testing is cheaper than continuous O(n^Z) testing.

18
  • 5
    Unit tests and integration tests are not alternative means to the same end, they serve different purposes. This is like asking "I want to write a novel, which is better: vowels or consonants?" Commented Oct 23, 2023 at 13:59
  • 3
    @MisterJeps I need an academic paper that shows that academic papers trump books written by people who code for a living. Commented Oct 23, 2023 at 14:26
  • 3
    You won't find an "academic paper" for this because your maths is badly flawed, or at least overly simplistic - the whole point of integration testing is to help check there are no unexpected interactions between f and g. Sure, you can assume there are no interactions, but then the argument becomes circular. Commented Oct 23, 2023 at 14:32
  • 3
    Just because f works and g works doesn't mean that f combined with g works. Only doing the unit tests might be less work, but it also is less evidence that your code is correct. And I suppose there are plenty of systems where you have no "expected result".
    – gnasher729
    Commented Oct 23, 2023 at 14:37
  • 3
    f and g can be part of the same unit. The idea that a unit is only the smallest thing is simply false. f(g(x)) is perfectly unit testable provided g isn't the database. Commented Oct 23, 2023 at 14:39

2 Answers 2

5

I am not convinced that unit testing beats integration testing hands down in terms of man hours.

Unit testing is not something that can beat integration testing. Unit testing is an effective way to test a unit of code that happens to be deterministic and isolated thus making it suitable for unit testing. The audience for a unit test is very much a coder working inside some unit of code. Likely one who just typed a semicolon and wants to know if that last line broke anything. This test should be fast, deterministic, parallelizable, and run often.

Integration testing is not something that can beat unit testing. Integration testing is an effective way to test the configuration of a system and that it's parts interact as intended. The audience for an integration test is the merge tool. This provides more time to run these slow tests that wont be run as often.

Acceptance testing is not something that can beat integration testing or unit testing. Acceptance testing is a way to map requirements to tests at a high level that focuses only on behavior. This test proves that the work done satisfies what was asked for. The audience for Acceptance testing is the product owner. They can be slow and abstract and run as rarely as demos are done.

Not one of these replaces the other. Sure they can run the same code. That isn't the point. There is more to testing than code coverage. There's who cares about the test.

So, if you're arguing that you shouldn't leave out Unit Testing just because you can Integration Test I'm with you.

If you're arguing that you should leave out Integration Testing just because you can Unit Test I can't help you.

3
  • i'm arguing that we need both, yes. but the people i argue with insist we only need system testing. Commented Oct 24, 2023 at 14:05
  • @MisterJeps I’ll let you in on secret. You can get a lot of the benefits of unit testing without telling anyone. Write your own tests on your own project. Run them yourself. They will help you as you write code. If others see no value in them it’s their loss. I know they help me. Commented Oct 24, 2023 at 19:48
  • So I've had it confirmed that (1) system-level testing - testing the whole system at one - scales as the SQUARE of the number of features. And (2), UNIT tests scale LINEAR with the features. I suspected this was the case, just wanted to get some math behind it. It's not an opinion, it's a provable fact. Doing unit tests reduces the set of system tests required, so you need both but the work is reduced. Finally. Thanks everyone! Commented Jan 3 at 18:18
3

If we run the system and get the expected result, everything must be working ok

The fallacy of this is that it becomes very cumbersome to achieve the same level of test coverage.

Unit tests excel at testing isolated components of moderate to high complexity. Since the tests are done in isolation it is very easy to cover many different scenarios.

Also note that "unit" may represent a component of arbitrary size. It can be a single class, but it might as well be a service composed of hundreds of classes. As long as there is a well defined interface and behavior it can be considered a "unit".

Without unit tests you take several risks:

  1. Bugs may be undetected for far longer, since no integration test used the component in the particular way that triggered the bug. It is perfectly possible that no integration test covers the component at all.
  2. If you find an issue it is far more difficult to know what component is responsible.

An real world example where I find unit testing super useful is a KD-Tree used for finding closest points. There are no significant dependencies, so tests are easy to write. There is plenty of chances to make fairly simple mistakes, many that only appears in specific situations. And it is very easy to confirm that the results are correct. Achieving the same coverage with an integration test would be challenging to say the least.

In conclusion you should write whatever form of testing are most useful to you. Some classes are mostly "glue", so benefit most from integration tests. Some are mostly logic and benefit most from unit testing. And in some cases it is useful to combine many classes into a larger "unit" and test that just as you would with a single class.

1
  • Thanks Jonas, unfortunately the question was not in a correct format so it's been closed, but i appreciate the input. Commented Oct 23, 2023 at 20:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.