4

A follow up question to How do unit tests facilitate refactoring without introducing regressions?.

I said that integration tests test the behavior of the code, while unit tests, being tied to particular classes and methods, instead test the way it is written. Several answers & comments, if I understand them correctly, challenged this assumption. They seemed to believe that if I write a unit test against the public surface of a class then (even if all dependencies of the said class are mocked?) I am testing the behavior of the code, and not the way it is written. I would be testing the way it is written if I, instead, insisted on testing private methods or write assertions of the sort "assert that method X calls method Y, then method Z".

I find it somewhat surprising - I've always thought about individual classes (and therefore also, in particular, their public surface) as just an implementation detail of the entire application.

This matters because I claimed that unit tests need to be rewritten if refactoring occurs, while others - as I understand them - said that this is only true if public interface of a class changes, which happens much rarer than changes of its implementation.

One of the benefits of unit testing, as per proponents, is that it enables or forces a very specific architecture where all methods and classes will be very small. For methods, as per Uncle Bob, this means they should be generally not longer than about 4 lines. For classes this means the Single Responsibility Principle. It is often said that private methods are classes waiting to be factored out, which implies that classes should, generally, have few private methods.

Such an architecture that focuses on tiny methods and classes that come in large numbers seems to me to make it even more true that unit tests test the way code is written and refactoring must invalidate already written unit tests. If methods are extremely tiny it will be rare to limit our changes just to this particular method and this method only. Refactoring will also only rarely be limited just to the implementation of a single class without touching its public surface. Rather, refactoring will rearrange already existing methods and classes, modifying their public surface (and invalidating unit tests written against this public surface). Indeed, one of the most common operations will be splitting a class or a method in two. This will very obviously invalidate the previous interface of a split class (or method), necessitating rewriting of already existing tests.

Am I failing to see something obvious when I treat individual classes as merely an implementation detail? How does the public interface of individual classes and public methods stay constant during refactoring if there are very many very tiny classes and methods? How is writing unit tests for the public interface of each of such tiny classes not testing the way code is written, but rather just testing for its behavior?

8
  • 8
    I've read each of your questions, recently, and I've noticed a general misunderstanding of what constitutes a unit test, why people write unit tests, and how best to actually structure code to facilitate testing. It is difficult to answer this question, because there are so many misunderstandings. It almost sounds like "unit testing" has been pounded into your head by others without fully explaining why. Given the effort you've made to write thoughtful questions (despite this community's propensity to down-vote everything), you really seem to be struggling with this. Mar 20 at 19:44
  • 2
    Can you edit the last paragraph of your question to focus on a single perspective of this? We have too many questions in one post to make this answerable. Mar 20 at 19:48
  • @GregBurghardt Ugh... I'm thinking about it and I'm not sure how to tackle this. The last paragraph has 3 questions. The 3rd one is what I care about, but I need the 2nd one to state the 3rd one. In retrospect I should've probably asked two different questions: (1) does an architecture that follows SRP and prefers very tiny methods mean that classes and methods will be rearranged very often? and (2) in light of (1), are unit tests not fragile? I should probably cut this question to (1), but I can't because it already has two answers covering (2)
    – gaazkam
    Mar 20 at 21:39
  • 2
    To be honest, the SRP and the number of lines in a class or method is completely irrelevant for unit testing. And remember those are guidelines for designing modules in your application. Use them when they make sense. Discard them with prejudice when they don't. Mar 20 at 22:13
  • I think the questions in the last paragraph are related enough, to be honest. Mar 20 at 22:19

5 Answers 5

4

Does testing the public surface of a class test the behavior of code or the way it is written?

Yes.

It depends on how the test was written. A good test is defined by the the System Under Test and the assertion being made about that system. Consider a unit test for a method that returns the result of some calculation. If you Assert.Equals(42, total) then you are testing the behavior of whichever class or method calculated the total. If instead, you call the same method in your test, but you assert that a method in your mock repository object got called, I would argue the test is verifying how the code is written.

Using the hypothetical example above, let's also assume a repository or data access object is necessary as part of this calculation. Your test could look something like:

var mockRepository = new Mock<ISomeRepository>();

mockRepository.Setup(mock => mock.GetValues("foo"));
              .Returns(5);

var calculator = new Calculator(mockRepository.Object);

var total = calculator.GetTotal();

Assert.Equals(42, total);

The mock repository is necessary because it is a dependency of the System Under Test. The mockObject.Setup(...) line sets up a method call with the predefined return value necessary to verify the expected total.

Now it comes time to refactor. The GetTotal method is a hot mess, and you decide to pull some of the logic out into its own class. Now the Calculator class has two dependencies: the repository, and this other class that contains some utility code. Guess what? You need to refactor the test too:

var statistics = new StatisticsCalculator(); // <-- the new dependency
var mockRepository = new Mock<ISomeRepository>();

mockRepository.Setup(mock => mock.GetValues("foo"))
              .Returns(5);

var calculator = new Calculator(mockRepository.Object, statistics);
//                              new constructor param: ^^^^^^^^^^

var total = calculator.GetTotal();

Assert.Equals(42, total);

Yes, you added a statistics object to your test, but crucially, the call to GetTotal() and the assertion remain unchanged. This is still a refactoring job, yet you needed to change the test. Yes, the test mocks a repository object, but that is a means to an end: asserting that the total is 42.

Don't confuse code structure with behavior. The test is still asserting that the total is 42. The code structure changed, so the test changed to reflect that, but the behavior of the System Under Test and assertion remained unchanged. You are testing behavior.

Now consider a slight variation to illustrate when you are testing how code is written:

var mockRepository = new Mock<ISomeRepository>();

mockRepository.Setup(mock => mock.GetValues("foo"))
              .Returns(5);

var calculator = new Calculator(mockRepository.Object);

var total = calculator.GetTotal();

Assert.Equals(42, total);

mock.AssertMethodWasCalled(mock => mock.GetValues("foo"));
//   ^^^^^^^^^^^^^^^^^^^^^

The last line where it asserts a method on a mock object got called is testing how the code was implemented. There are reasons to do this, but generally you don't need to assert that methods on a mock or other dependency got called. The test should only fail if the System Under Test didn't give you the expected result. Mocks exist to make a test deterministic when the System Under Test has potentially volatile dependencies. Mocks exist to control the System Under Test such that failures can be attributed to the behavior of the System Under Test rather than an environmental factor you cannot control.

The way you write a unit test affects whether you are verifying behavior or the way code is written. Typically you want to verify the behavior of the System Under Test and not which methods on which dependencies got called. More importantly, refactoring != don't change the tests. Code structure can change, making changes to your tests necessary. The conceptual idea of what you consider to be the System Under Test and the assertion should not be changed.

Think of it another way. Every test is organized into 3 phases: 1) Arrange; 2) Act; and 3) Assert. Changes to the "Arrange" phase of a test are common during refactoring, and do not indicate you are testing the way code was written. If you refactor code and need to change the "Act" or "Assert" part of a test, then you are not refactoring. If you need to change the Assert when refactoring, consider for a moment if you have found a bug, or a meaningless assert.

Asserting the total equals 42 is good. Asserting that a method got called on a repository is pointless when the output should be 42. In fact, you could set up a fake method call to your repository, and if the GetTotal() method no longer calls the repository, your test should still pass without needing to modify your asserts.


Now, if you have this repository, and nobody is calling it, consider whether you need the repository, but that is a design decision out of the scope of this question, but unit tests can point you to unnecessary dependencies. Comment out a stubbed call to a mock object and the test still passes? Maybe that mock isn't necessary anymore, and its worth some analysis to see if you can remove that dependency.

2

I apologize for this being a long form answer, but the very topic of your question is one that yields benefits to you in complex business cases. Therefore, to highlight the purpose of the advice you've been given I wouldn't be able to showcase it using a very short and simple example.

if I write a unit test against the public surface of a class then (even if all dependencies of the said class are mocked?) I am testing the behavior of the code, and not the way it is written.

It might seem like a bit of a nitpick, but the feedback you're getting about testing is assuming that you already designed your class in a way that its contract (i.e. what you call public surface) is abstracted from its implementation details, in a way that the contract accurately represents a well-defined behavior (disconnected from any knowledge on how the logic to achieve this behavior would need to be implemented).

If you're currently not designing your classes like this, then you're going to see a discrepancy between what the advice says and what you're seeing.

Partly, I'm going to blame the advice here for being opaque and self-referential. It doesn't make sense to assume that your class is already designed like this, because if you were doing that already, you probably wouldn't be asking the question you're asking right now.

So let's elaborate on that hidden part of the advice you've been given.

I've always thought about individual classes (and therefore also, in particular, their public surface) as just an implementation detail of the entire application.

The short answer here, which will require elaboration, is that not all classes have "public behavior" in the sense that it is used to describe unit testing methodologies.

A unit can be composed of multiple classes which interact with one another, but together they represent a single component. That interaction between these classes is considered a private implementation detail, even though some of it relies on the public keyword in order to allow these classes to interact with one another.

For the purpose of describing the behavior that we test, there is a difference between what is public to a class and what is public to the unit under test. Unit testing methodology focuses on the latter, and it does not care for the internal communication happening between individual classes of the unit.

For example, let's say we are building a vending machine. We need to obviously confirm that the machine we've built works before we ship it to the customer. We could test the following:

  • We put a coin in the machine.
  • We select the drink we want.
  • We receive the drink we selected and paid for.

This is the equivalent of an integration test.

But we would also like to test individually if the machine is able to (a) receive money, (b) understand the customer's selection, and (c) dispense the respective drink. These are the unit tests. The reason we want to be able to test these individually is if our integration test fails, we want to know which part failed, which is why we need these unit tests.

For the purpose of the example, I'm going to focus on (b).

Notice that I described (b) as understanding the customer's selection, not as "the pressed button sends an electrical signal that matches the drink identifier". The latter is a technical implementation detail. Yes, that is what happens under the hood, but "under the hood" means private, and this behavior is described in a way that reflects its public expectations.

At the end of the day, what I want to test is if this scenario is correctly handled:

  • I press the Sprite button
  • [magic]
  • The vending machine is aware I want Sprite, not Coke or Fanta.

Early foreshadowing: notice that I've taking the original second bullet point, and I have further subdivided it into sub-bullet points.

[magic] is not a single thing. Depending on how deeply you investigate, there are a lot of parts that interact with one another to make this happen.

  • The physical button is pressed, which makes an electrical contact that was not there before, which creates a signal
  • Releasing the button also must break that contact, it should not stay on indefinitely.
  • That signal travels along a conductor (i.e. electrical wire), and it reaches the controller (i.e. the thing all the buttons are hooked up to).
  • The controller is able to discern which button is sending a signal, find the related price of the product, compare the price and the money entered into the machine.
  • If successful, it sends a different signal through to the correct dispenser (because a specific button should map to a specific dispenser).

But a lot of these are technical implementation details. For example, it doesn't matter that we're using a physical button. We could be using a touch screen, an NFC reader, a voice assistant, ... and the process would be the same. We might currently be making selections using buttons, but we need to design our test around selections, not buttons specifically, because we're interested in the public behavior (the ability to select), not the specific implementation (physically pressing a button).

A more correctly abstracted test design would be:

  • Arrange
    • Unit under test - We instantiate all of the components that are necessary to convert a drink selection to a dispenser signal, i.e. button, wire, controller; and we wire them up the way they're supposed to be.
    • Mocked state - The price of the item is $2
    • Mocked state - The money counter component is telling you that the customer has put $2 in the machine
    • Mocked spy - A fake dispenser is made, which is going to report back to us which drink it was told to dispense.
  • Act
    • We select Sprite
  • Assert
    • The dispenser is told to dispense a Sprite drink.
    • The dispenser is not told to dispense any other drinks.

If tomorrow we change to using a touch screen, we're going to have to make changes to the "unit under test" section, because that's inevitable. This step instantiates the real components, so any change made to the components being used will obviously require updating.

But the test remains the same for everything else. That is the point.

If your test had been designed around the concept of a button, the entire test would need to be rewritten. Instead, by designing the test around the concept of a selection (which currently happens to be done via a button), we are able to swap that button out with minimal impact to the test.


To conclude, I want to refer back to the thing I already foreshadowed above:

Early foreshadowing: notice that I've taking the original second bullet point, and I have further subdivided it into sub-bullet points.

Taking a complex orchestration and breaking it down into simpler constituent parts is the essence of software development, and more importantly it is a recursive process. You can keep applying this to subcomponents, and you should keep doing this until each individual subcomponent is trivially definable, implementable and testable.

If you're struggling to define it, implement it, or test it; you probably need to break it down further.

But you need to know when to stop. You can always break things down further, but it's not going to keep making things easier. At some point, further decomposition starts making things more difficult. This should be avoided. It is a form of dogmatic programming, i.e. decomposing things for the sake of decomposing them, not for the sake of it actually having a practical benefit.

When you said this:

I've always thought about individual classes as just an implementation detail of the entire application.

It sound like you're immediately doing a full decomposition in a single go, i.e:

The whole application => specific classes

But it would be more helpful to do this in individual steps, e.g.:

The whole application => a bounded context => a component => one or more specific classes

The amount of steps there are is not universally answerable, it very much depends on your context. Some applications are really simple, some are really complex. This obviously changes the amount of layers there are.

Generally speaking, the implementation lives on the last step in this chain (i.e. "one or more specific classes"), but the definition of a "unit under test" tends to live on the second to last step in this chain (i.e. "a component").

This disconnect between the level on which your write your implementation and the level on which you design your test, I suspect, is the core part of what led you to post this question.

If you do the decomposition step by step, you'll be more attuned to how the initial big picture (which describes your business requirements) slowly morphs into your specific implementation details, and this will be the indicator that helps you identify what should be labeled as a "unit" for the purpose of "unit testing".

If you don't decompose it enough, it's more an integration test than it is a unit test. If you decompose it too much, your unit tests will actually become "class tests" that are so intwined with your implementation details that you're going to have to continually rewrite the tests, which is where the process is no longer beneficial to you.

I don't know your specific experience level, but finding that line is very difficult. Experienced senior developers still struggle to sometimes clearly identify these lines. Don't worry about getting it objectively correct. Just try to do better when you realize that there's an imperfection.

23
  • 1
    "We could be using a touch screen, an NFC reader, a voice assistant, ... and the process would be the same." - the same for who or what? I find it very hard to believe that a test harness designed around the idea of a keypad entry (that being the public surface), would work equally well with a machine expecting voice input at the public surface. The only thing that is the "same" here is the conceptual idea of an item selection (i.e. the similarity exists purely in the human mind) - there's nothing the same mechanically about the interface under test.
    – Steve
    Mar 21 at 6:07
  • "The reason we want to be able to test these individually is if our integration test fails, we want to know which part failed, which is why we need these unit tests." - I've seen people using this argument to support the idea that each class and each method (maybe even private methods) must be specifically tested (if I understand these people correctly). A unit test must test a single unit of code, because if a unit test fails, then we must know precisely which part of code is erroneous, with no further debugging. (cont)
    – gaazkam
    Mar 21 at 8:38
  • So a failing unit test should, ideally, point me to the precise method that needs fixing. Obviously such tests are not sufficient, so on top of unit tests we must have component level tests. If all unit tests pass, but a component level test fails, then we know the error lies not in any particular method, but rather in the interaction between methods within a single component. On top of component level tests we must have integration tests and on top of this we must have full end-to-end tests. (cont)
    – gaazkam
    Mar 21 at 8:42
  • But tests should be pushed downwards: if something can be tested by a test that lies lower in Cohn's pyramid of testing, it must (so that a failing test at level X shows there is an error in the way components at level X+1 interact, and not in a component at level X+1, because this would mean a test at level X+1 would have to fail instead). In this way I need to have a very comprehensive test suite that reflects Cohn's pyramid of testing.
    – gaazkam
    Mar 21 at 8:45
  • To be clear, I don't like the above advice, mainly because writing such an extremely comprehensive and multi-level suite of tests seems to me to be extremely costly and annoying to write, hence my current stream of questions...
    – gaazkam
    Mar 21 at 8:49
1

From a certain perspective you are right: Unit tests test internal parts of an application and are therefore always coupled to implementation details. If the entire internal architecture of the application is rewritten, all unit tests would become obsolete - even if the external behavior of the application remains the same.

But consider a code base developed strictly following the Open-closed principle. In such a project unit tests would never become invalid because public interfaces are never changed, only extended. You would need to add unit tests when adding new functionality, but you would never need to alter existing unit tests.

In reality, most project does not follow the open-closed principle strictly, and allows changes to public interfaces when it is deemed to improve the overall architecture. In those case you would have to change test and code at the same time, which is always risky. But if there is good test coverage, there will be a level of redundancy which should catch those bugs.

For example, imagine you have a class A which depend on class B:

 [ A ] -> [ B ]

If you follow test-driven development, you would have tests for both A and B. But imagine you have to change the public interface of B. You would have to make code changes to both A and B and modify the tests for B. But the tests for A would not need to change, which means they should catch any bugs introduced by the change. So as long as you are able to change one class at a time, you can safely make changes to public interfaces.

Your assume that you would often have to make changes to the public interfaces many classes at the same time. I don't think this should be the case if the code is loosely coupled and generally follow good design principles. The ideal is to make many small changes incrementally and having all tests be green between each change.

6
  • "a code base developed strictly following the Open-closed principle" - I think you're misunderstanding that. Meyer (who was writing over 35 years ago) did not devise a method of software design that, if followed "strictly", guarantees no design error and no future re-adaptation in the system of interfaces. Rather, he was describing principles or ideals to which interface designs should conform. But that's begging the very question of achieving those ideals - we change the system of interfaces because we find it is not ideal, or has ceased to be ideal as circumstances have changed. (1/2)
    – Steve
    Mar 23 at 18:16
  • "The ideal is to make many small changes incrementally and having all tests be green between each change." - it would certainly be ideal, but I find often that even "small" changes can require tweaks all over a codebase, and that's before even considering real architectural upheavals when something originally quite simple has to be revamped to cope with a considerably more complicated new requirement. I don't think any development method is living in the real world when it assumes changes are only ever incremental, local, and that the software really works after each increment. (2/2)
    – Steve
    Mar 23 at 18:39
  • @Steve: The open-closed principle is strictly adhered to in for example the .net base class library in order to not break backwards compatibility even across major version. AFAIK they only accept backwards compatibility breakage for security issues. Of course this does not mean the BCL does not have any mistakes or examples of bad design, it just means that they can't fix design mistakes by altering or removing public interfaces, only by adding new public interfaces. Classes and method may be deprecated when there is a better alternative, but are not removed.
    – JacquesB
    Mar 23 at 19:14
  • @JacquesB This approach is understandable, but it does have its downsides: mess will gradually accumulate. Programming language standard libraries may have no other choice, but the situation in C++ clearly shows what this eventually leads to.
    – gaazkam
    Mar 23 at 20:04
  • @gaazkam: Absolutely, and I will not recommend this approach for the typical application development. I mention it is an example of how it is possible to develop and refactor without ever having to alter existing unit tests. For most application development it is better to change public interfaces when appropriate and update tests as necessary. But you shouldn't have to update both many interfaces and many tests as the same time, this is too much risk.
    – JacquesB
    Mar 23 at 20:21
-2

You assert that refactoring will modify the public surface of classes and hence break unit tests, but don't say why that won't break integration tests.

An integration test is just a unit test with less mocking.

Even an end to end test, where you say test "when i click on the button a thing happens" is obviously going to break if you move the button.

Instead of thinking of individual classes as implementation details of an application, think of them as library classes that perform an operation you are using in your application. You never change the interface to Math.SquareRoot, you just assume its always going to work. Why do your treat your own classes differently?

14
  • 1
    "but don't say why that won't break integration tests." - in his previous question, he did in fact explain, along the lines that because integration tests have more internal mass and less surface area hooked by tests, it's more likely that an alteration would be purely internal and wouldn't therefore affect the tests. Not endorsing that logic, but just stating it. (1/2)
    – Steve
    Mar 20 at 21:27
  • "You never change the interface to Math.SquareRoot, you just assume its always going to work. Why do your treat your own classes differently?" - I would suggest it may be because the workings of basic mathematical operators have been settled over centuries by people who spent their whole lives in mathematical study, are highly general, and are tested in use by billions of users. Whereas the same cannot be said about the design of the average application codebase - much of which will have been written this year by one person with next to no experience of anything. (2/2)
    – Steve
    Mar 20 at 21:29
  • to me the previous question doesn't really cover the difference, just asserts that there is one. The reason you don't change the interface on sqrt isn't because the definition is settled, (your average sqrt doesn't handle complex numbers) its because you treat the library as a completed component of your application. My point is that you should treat your own application components, classes, in the same way. Then the answer to the question is clear.
    – Ewan
    Mar 20 at 21:34
  • I don't follow that logic myself, because the system of components and classes in an application is not necessarily fixed or "completed". I don't see much of a natural analogy between the relative stability of standard mathematical tools and operators, and the stability of things which are written bespoke for specific business applications.
    – Steve
    Mar 20 at 23:00
  • I'm not saying there will never be any refactoring, you might change your Maths library and have to refactor your code to a new interface, you might have a higher chance of having to change the interface to myComponent. But if you treat components as components instead of my code and external code, then you don't have this question of unit tests being implementation tests. They are behaviour tests for a component
    – Ewan
    Mar 21 at 10:26
-2

By definition, refactoring changes the way code is written without changing is outside behaviour. There’s a famous book called “Refactoring” and thats his authors definition. Martin Fowler his name.

You can make very useful changes that intentionally change the outside behaviour and therefore break init tests, but that is not refactoring.

Now a class can have private methods, not part of the interface, so they should have no unit tests. You can change them and it is still refactoring- obviously only as long as callers that are unit tested are changed accordingly.

And remember that at some point you are actually implementing your module, and while your private methods are irrelevant to users of your module, they are very relevant to you as the author of the module. So you may very well have private unit tests, so that any user of your private methods (in other words, you and the module and anyone taking over maintenance of your module) can rely on your private methods.

4
  • 3
    I'm not 100% sure if we agree on definitions. Assume I make changes to some classes in such a way that the behavior of public methods of these classes changes. However, the behavior of the entire application whose code contains those classes does not change. Is this refactoring for you? I'd call this 'refactoring', but I start suspecting that others might not call this refactoring on the basis that public surface of some classes changed.
    – gaazkam
    Mar 20 at 21:26
  • What is the scope you are looking at? If you are looking at the scope of the application, it is Refactoring. If you are looking at the scope of the class, it is rewriting. By the way, I notice a lot of emphasis on classes in your questions and comments. I would like to remind you that object-orientation is about objects, not classes. In fact, OOP isn't really about objects either, it is about messaging, i.e., what happens between the objects. Likewise, you seem to be under the impression that unit tests test classes or methods. They don't. They test units (of behavior). Mar 20 at 22:10
  • @JörgWMittag, but if refactorings don't require tests to be changed, and rewritings do, then how can it be entirely in the eye of the beholder? Either the tests actually have to be changed, or not - it can't depend purely on how the programmer thinks. And if "units" are not methods or even classes, but something as large as a module with its own defined external interface and consisting of many classes internally, then it's fair to acknowledge that many practitioners do end up testing individual classes and methods in isolation, not just whole modules.
    – Steve
    Mar 20 at 23:16
  • 2
    @gaazkam: In your example, that's generally not what's called "public" in relation to testing strategies. If your application still behaves identically, then the public behavior hasn't changed, by the very definition of what public behavior is. I think you're conflating the public keyword in a class with the "public" in "public behavior". They are similar but not the very same. The "public" in "public behavior" is application-scoped (= integration tests) or component-scoped (= unit tests), it is not inherently class-scoped. It's only-class scoped when a component is made up out of one class.
    – Flater
    Mar 21 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.