What is the point of unit tests?

Question

I've been a software developer for 20+ years, although it's been an unusual career. I've mostly worked on either legacy projects, or small standalone, non-public-facing apps and so only a couple of times has it been mandated by my employer to include unit tests in my work.

To be honest, I struggle with the whole concept. I'm now working on (for me) a very large and complex system with thousands of unit tests ... and yet I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test. So, in essence, the unit test discovered that it wasn't in sync with the code it was testing, but that's all. Never in my experience has a unit test failed because the unit under test wasn't doing what the unit under test was contracted to do.

I've asked numerous colleagues why we bother with unit tests, since an enormous amount of time and effort goes into creating them but without (so far as I've been able to tell) any discernible business benefit. Our codebase has its share of bugs, but they are all the result either of inadequate requirements/acceptance criteria, or poorly written business logic. My colleagues answer my question with a kind of blanket "mom and apple pie" - of course unit tests are good, because everyone says they're good; that's the accepted wisdom.

My other gripe is that when I have to write them it seems that frigging the setup (mocking repos, interfaces etc.) is just as likely to be error-prone as the test itself, and the hours spent creating the tests is time that could better be spent on fixing known bugs, refactoring code, addressing technical debt etc.

I really want to be convinced, by the way! I have looked for answers to my question but so far I haven't seen or read anything remotely compelling. Nor am I convinced by appeals to authority, so simply saying "Martin Fowler" won't cut it with me.

Answer 1

It's easy to write tests that have no value.

Especially if your employer mandates them. "You wanted tests. Here's your tests." If that's your criteria for tests then you're exactly right. The only thing they're doing is appeasing your clueless employer.

The ability to write useful tests is a hard-won skill. It is not something that emerges easily when writing tests just because you were told to. You need to have a use for them in mind and do work that proves if they actually are useful.

It doesn't help that there are different philosophies about how it should be done. Some want every class isolated and all its dependencies mocked. Others want only high-level tests that show some behavior exists in support of some requirement.

I've come to the conclusion that, much like documentation, the most important thing to consider is a test's audience. A test can do a lot of damage if it was meant to prove a function's behavior that turns out to be replaceable with a completely different function with a different behavior that can't pass this old test. Such tests are fine in the hands of those that can see that and know to discard the test. But when others get hold of that test and won't let it go, it makes refactoring harder, not easier.

There are tests meant to satisfy your product owner. Tests meant to satisfy QA. Tests meant to teach new coders how the API works. Tests that enable refactoring without changing behavior meant to satisfy your peer reviewer. Tests meant to prove this implementation happens to behave the way you intend which should never be shown to another coder.

Why? Because while the test makes sense to you, it's weird and sure to confuse others. Tests that confuse people never die. Not every test is ready for public consumption.

Many argue about what to call these tests. Unit tests, integration tests, acceptance tests, end-to-end tests, automated tests, regression tests. These terms have become meaningless to me because I've seen them used to mean so many different things. I've learned to just keep asking what the term means in this particular shop until it’s clear. I want to know the point of each test. I don't blindly trust the names to tell me that anymore. Burned too many times. "It's a unit test" "Why?" "Because we run it with jUnit" "Ugg"

You're stuck with a testing requirement and no clear vision of the point of that requirement. I've been there. It's tough. If you have the time, try to identify the most useful thing you could do with testing for your employer. Put your focus there before trying to master everything.

TDD (Test-Driven Development) is an art. You really need to practice at home. TDD done halfway makes a nightmarish mess. You'll know when you have a handle on TDD when you do it, not because you're supposed to, but because it makes you faster. If it's slowing you down you're still not ready to do it at work.

Testing for planned refactoring is another story. Michael Feathers' book Working Effectively with Legacy Code is a must-read if you want to transform a code base with no tests into something else that has some tests that prove that you only changed what you meant to change.

Shops abound where the only point of tests is to have tests. Figure out something truly useful to do with them, or your shop will turn into yet another one. If so the tests will slow you down and cripple you.

Probably the best thing you could do is make the point of each test clear enough that those who come after you will have no doubt when it's time to delete the test. This is critical because when in doubt, coders tend to drag old useless tests around.

If the test exists in support of some requirement, say that. If the test exists because some class exists, say that. If the test exists because some API exists, say that. This lets me know when to discard the test.

So when you ask, what's the point of unit tests, my answer is nothing until proven otherwise. If you can't figure it out then it needs to die. Clutter is not helpful.

Good tests show expected behavior. Good tests show intent. Good tests show what wasn't thought of. Good tests show how to express what wasn't thought of. Good tests are a handy way to peer review code. "Shouldn't your code do this? Because it doesn't."

You know a test is good if, when it passes, it makes you confident. You feel no need to read the code it covers. You trust that code the way you trust your languages print command.

But just because a test passed does not mean it’s a good test. Be suspicious of tests you have only ever seen pass. Trust tests you’ve seen both pass and fail. That’s how you test the test.

Answer 2

Sorry that this answer is so long, but your points are well constructed and require details and nuance to counter. If you're looking for a TL;DR, I think the last section is the most important one for your specific circumstance.

Testing your own work is prone to bias

Never in my experience has a unit test failed because the unit under test wasn't doing what the unit under test was contracted to do.

Who is writing the unit test? The developer who wrote the implementation? Yeah; then that's exactly what you're going to get.

Let's say I'm asked to implement a factorial calculation (!) but I misunderstand the requirement and think its about addition (6! = 6+5+4+3+2+1) instead of multiplication (6! = 6*5*4*3*2*1), then I'll obviously write a wrong implementation. But if I design the test as well, then I'm going to write tests that confirm that the resulting value of 6! is 21.

The only thing a dev can catch when they write their own tests are issues where they mistakenly wrote code that does not do what they think it does. But anecdotally I can tell you that this is not how most bugs originate. It is far more common for developers to make misinformed decisions, whether that is due to a misunderstanding of the requirements or of good practice guidelines.

A significantly more reliable way to catch misinformed decisions is by having the test cases designed by a different person than who designed the implementation. Regardless of who is wrong, if the tests disagree with the implementation, this alerts the dev (and tester) to find out why they're disagreeing.

The danger of regression

I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test. So, in essence, the unit test discovered that it wasn't in sync with the code it was testing, but that's all.

I already addressed that this is inevitably the only thing your tests tend to do when your developers write their own tests. However, I also want to address the undertone of your question that asserts that this behavior is not valuable, because I am of the opinion that is very valuable.

The necessity of tests is only visible when you are suffering the consequences of not having any. I've worked on legacy codebases that were a game of non-stop whack-a-mole. Whenever you fixed any issue, you'd invariably create another issue somewhere else. Every time I made a change, I would live in fear of what else was going to break because of it. This is an incredibly stressful environment to work in.

What I'm describing here is exactly the same as when you say:

the developer changed the unit under test

The benefit that you have, which I didn't have in those past projects, is that you have tests to tell you exactly what you changed. Without that test to alert you, if you cause something to change that you weren't aware of, you wouldn't be made aware of it.

Tests flag issues when they occur, and they prevent issues from being unseen and piling up until you get to a tipping point where there's so many issues that it becomes impossible to tackle them without creating new ones. Therefore, the existence of tests completely hides the problematic consequence from not having tests.

You've already been referred to the preparedness paradox, and this is exactly what you're succumbing to. You're already sitting on the greenest grass and are therefore subconsciously not valuing the work that went into making sure you'd be sitting on greener grass.

Better to fix things that prevent them??

the hours spent creating the tests is time that could better be spent on fixing known bugs, refactoring code, addressing technical debt etc.

Strong disagree. I cannot overstate how much I disagree and have personal anecdotal evidence to back this up.

I think you're making a biased observation from a position of a reasonably good developer experience - whether this is because of high developer skill, focus on good practice, decent software development budget, or strong test coverage; is unclear to me. Regardless, I can count at several projects I've worked on in the past where bugs that should be trivial would take months to actually fix.

I started at a company where they said "this project is pretty much done, 5 more bugs and it's delivered". The requirements never changed, but 18 months later the project was still "almost done", now with 21 bugs, and two senior consultants had already been brought in to do major refactors.
In hindsight, it would have been faster to throw out the entire codebase and start from scratch (from the point where they hired more experienced developers to lead the dev team). This is how bad things can get.

This is why I can tell you hand on heart that the pre-emptive effort of writing tests feels like a lot of effort but that's because you don't see what kind of future headaches you will have prevented.

If you were to create two timelines, one where you test (A) and one where you don't (B), and then measure the time it took to write the tests (A) and the time difference in bugfixing (B minus A) to get the application to the same level of quality, B will have taken more time by several factors.
I am not a time lord and can therefore not show you the same project in both of these scenarios, but I have over a decade's worth of anecdotal experience as a consultant (who was often brought in to projects on the brink of failure to fix things) to back up my assertions here.

The difficulty of writing tests

My other gripe is that when I have to write them it seems that frigging the setup (mocking repos, interfaces etc.) is just as likely to be error-prone as the test itself

I'm ignoring possible explanations like low developer skill or lack of testing experience as those issues have trivial solutions (education, experience, mentoring).

You should use this as an early indicator that your contracts between your components are too contrived.

If you think of your application space as a blob of logic, and we are in the business of subdividing that blob into separate components, there are countless ways of making those divisions.
The better way of dividing things is so that the components themselves contain the complexity, and the contracts (i.e. what links the components) are as simple (i.e. not complex) as possible.

If you're in a situation where that's inverted, where the components are simple and their contracts are complex, your argumentation makes sense:

• Implementations are trivial, what's the value in testing them?
• Contracts are complex, tests are complex to write.

This causes friction for two reasons as mentioned above. It is of course possible that only one of these points is relevant to you, either because both your implementations and contracts are trivial, or because both your implementation and contracts are complex. I can't decide that for you.

I maintain that testing is beneficial to product quality. However, as mentioned in the above points, certain scenarios are easier and more justifiable to test than others. This shifts your responsibility. Not only should you be testing, you should also be designing your code to be as efficiently testable as possible, to ensure that you can at the same time reap the benefits of testing your code and still produce implementations are an effective pace.

Answer 3

You work on legacy applications. Have you ever run into some weird edge case where you have a function returning a weird value intentionally because a third party system depends on you returning a weird value?

You write unit tests to say that you expect the unit to return a given response with a given input. It is a way of documenting intent. You are saying that it is no accident that the unit is returning a value that might not be obviously correct. They are business rules consolidated to logical functions rather than comments.

As an example, I have a legacy application that wants a pipe delimited list of strings. But for whatever reason, it wants an empty list as "||". I can't change that application. If another developer comes in and sees the code, they might see that as a bug and attempt to fix it. If they do that, though, the unit test will fail, because I wrote a test specifying that I always want the empty string input to create an output of "||".

Answer 4

and yet I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test

That's the point: You are safe from regressions and can refactor/change the unit without being afraid of introducing new bugs.

Our codebase has its share of bugs, but they are all the result either of inadequate requirements/acceptance criteria

Unit tests actually help putting these criteria into code, especially when doing Test Driven Development. Have a requirement? Just put it into a test, and you force your code to handle it.

or poorly written business logic.

If the logic is contained to one unit, the test should have caught that bug. Once you find a bug like that, add a test, and voila, your system will never regress to that buggy state again.

frigging the setup

Yes, it can be painful to create the mocks. However, I experienced that being forced to write unit test also forces me to write testable code, which in turn is more maintainable: Usually you get clear dependency definitions and a stable interface as a result. Additionally, there are frameworks out there to help with mocking (and verifying those mocks).

Answer 5

What is the point of unit tests?

• To document the contract.
• To ensure the code fulfills the contract.
• To serve as an example of how to use a component.
• To prevent regressions.
• To discourage unnecessary coupling.
• To enable rapid refactoring.
• To improve the success of continuous integration.

I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test.

Passing tests should be the normal state of things. Even if a particular test never fails during its lifetime, its existence documents a contractual behavior and acts as reference code for others who want to use the unit.

If a unit test fails because the developer changed to code, then that developer violated the contract, and code using that unit might fail if it depends on the old contract.

If this was an intentional change--in other words, if the developer meant to change the contract--the developer should have changed the test. (In fact, it probably would have been best to change the test first.)

If it was an unintentional change, the failing test should have alerted the developer before it manifested in a bug elsewhere, breaking your continuous integration.

Never in my experience has a unit test failed because the unit under test wasn't doing what the unit under test was contracted to do.

The unit tests are the contract.

If the test was testing for some side effect that isn't contractual, then it's an unnecessary test that should be deleted rather than "fixed."

My other gripe is that when I have to write them it seems that frigging the setup (mocking repos, interfaces etc.) is just as likely to be error-prone as the test itself ...

In my experience, if it's hard or time consuming to write a unit test, then it's likely the design should be improved. I've almost never had to make a mock of an interface for a unit test. If the design is good and you have to create a mock, it's possible you're beyond the realm of unit tests. Integration tests are a different story.

... time that could better be spent on fixing known bugs, refactoring code, addressing technical debt etc.

I understand that it can feel that way at times, but maintaining your unit tests actually accelerates those other tasks, especially refactoring (which is a great way to address technical debt).

Refactoring is restructuring the code without changing the contractual behavior. Having a fast way to ensure you haven't changed contractual behavior is essential. It allows you to make design improvements very quickly and with high confidence that you haven't introduced new bugs.

Taking a (probably overly) simplistic view on known bugs: Either there's a bug in a unit not fulfilling its contract or there's an integration bug because a unit is depending on non-contractual behavior from another unit. A unit that passes all of its unit tests rules out the former. The latter involves either changing the implementation (to remove the dependency on the non-contractual behavior) or to add the expected behavior to the contract (by first adding a test for that behavior and then making it pass).

Answer 6

Alternative answer: Ground for refactoring

Aside from what was said in the previous answers, each project deprecates from the day it started. No matter how hard you try, unless there is no new development, every project eventually heads to the state the technical solution becomes obsolete, deprecated, and old.

Practical examples from a legacy project

• Removing unnecessary libraries: About 5% of the DAO layer was implemented using QueryDSL. The queries were not complicated at all, though the library added complexity to the project as no current developer knew how to use it. The library was not particularly friendly to Lombok as it was not configured at all to work together. As long as the tests were present, it was easy to rewrite QueryDSL into plain JQL/HQL without breaking anything, and the project also gets rid of tons of generated classes.

• Introducing new SDK features: The service layer contained a fairly large number of various transformations and business logic based on loops, grouping, conditions, reducing... Most of the functionality was rewritten into Java Stream API so the project was no longer dependent on various 3rd party libraries (at least some modules). The refactoring came hand-to-hand with the Java upgrade and the project became sexier for new developers to recruit to join as no one is happy to work with dinosaur code.

• Understanding existing code: The crucial part of the business functionality covers exchange rate conversion. This part was quite covered to understand what the module really did, though the code itself was an enormous mess. Further functionality was expected to be implemented in the soon future. The developers would have had no confidence in simplifying and understanding the code as long as it hadn't been covered. The refactoring was done, and the amount of code was reduced by half and commented on - all thanks to a decent number of tests.

• Discovering new bugs: Thanks to refactoring some covered parts of the code, it was found that both code and tests were written wrong for years. Though no critical mistakes occurred, no one bothered to report an error (testing department, customers...). The developers discussed with the business the correct behavior and implemented it in such a way.

• Identifying unused code: The project was bloated by the proprietary internal framework that was no longer maintained and provided functionality not used by the customer. The team estimated more than 25% of the code-base was unused. The missing unit tests helped to identify what was not used as the proprietary internal framework had close-to-zero tests and only the project-specific overridden parts were somehow tested. The project lost some weight and the build time has reduced.

---

^{Source: My close friend complaining over the years.}

Answer 7

To be honest, I struggle with the whole concept. I'm now working on (for me) a very large and complex system with thousands of unit tests ... and yet I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test. So, in essence, the unit test discovered that it wasn't in sync with the code it was testing, but that's all. Never in my experience has a unit test failed because the unit under test wasn't doing what the unit under test was contracted to do.

(emphasis added)

It sounds like you have some good unit tests there! Unit tests tell you what the code ought to be doing (this is why I always start a test method name with should). The code itself tells you what the code is doing. What you want is a 100% match between the ought and the is. (See the philosopher David Hume for why you can't get an ought from an is, nor an is from an ought.)

If you can change what the code is doing without documenting what the code ought to be doing, how confident can you be that the entire codebase is doing what it ought to be doing? You could have extensive human documentation in the form of comments throughout your code, but how many programmers read such documentation and always keep it up to date? When was the last time you read an out of date comment or some other out of date documentation? Unit tests get the knowledge what the code ought to be doing out of your head and into code. Unit tests are knowledge and documentation that the computer can run! Unit tests document how the code is supposed to work and also how to use it. Unit tests document edge cases that are hard to trigger but are very important to get right.

A well written unit test (I've seen some truly awfully written unit tests!) should only fail for one of two reasons: 1. what the code is doing has changed, 2. what the code ought to be doing has changed (and the tests were updated to reflect this). In either case you want to be notified about it and change the other to bring it into line. A failing unit test can cause questions to be asked. It's been doing X for the last 5 years, now it's doing NOT X. Is that correct? Should it be doing X or NOT X? These questions are very valuable to ask and get answered. Bugs that have never surfaced in Production can be caught this way.

How do you know you haven't introduced any bugs when you change the code? You could make a simple change or optimisation, and suddenly 10 tests start failing. This is your indication that you need to be more careful with your change. On the other hand, if your code is well covered, you can refactor the code to your heart's content. Sometimes really drastic changes can be made to the code without affecting a single unit test! I love it when I see those hundreds of green ticks telling me all my tests pass! It gives me confidence that I haven't broken something that I wasn't focused on. It also allows me to focus just on the change I need to make. I don't need to worry about whether some other part of the code that I don't know about might suddenly break. I've worked on complex code without tests, and it's a lot slower because you have to check all the entry points by hand!

Another benefit of unit tests is that they force you to write better code. Instead of a monolithic procedure that's hard to read and debug, you end up breaking the code into easily tested pieces. A 10 line method is easier to test and debug than a 500 line method. Unit tests encourage you to write modular, reusable code because it must be used by at least two things: the code itself, and the test.

My other gripe is that when I have to write them it seems that frigging the setup (mocking repos, interfaces etc.) is just as likely to be error-prone as the test itself, and the hours spent creating the tests is time that could better be spent on fixing known bugs, refactoring code, addressing technical debt etc.

Your project might be badly designed if your tests are hard to set up. If it was designed without unit tests and tests were an afterthought, it's almost certainly badly designed. I do recommend Michael Feather's book Working Effectively With Legacy Code, in which he talks about what methods can be used to get legacy code under test. When fixing bugs, refactoring code, addressing technical debt, etc., I definitely don't want to do that without tests in place giving me the confidence that I'm improving the system, not breaking it further! Bug fixes are a classic area for unit tests. Write the test with the desired correct behaviour, then run it before you change the code itself. It should fail (red light). Fix the bug, then rerun the test. If you did it properly, it will pass (green light). This is red/green testing. Always start with a red light, and don't stop until all red lights have been changed to green lights. If this code is ever broken again in same way in the future, the unit test will catch it (regression testing).

In short, humans are fallible. Computers aren't. If I repeat a task enough times, I will make a mistake or forget something. The computer never will. It will check every single test every single time without getting tired, bored, or creative, or making a mistake. The most embarrassing thing that happens to me professionally is that some bug gets by me into production code and is discovered for the first time by a customer. The number of bugs that I would have missed that my unit tests caught more than makes up for the time spent writing them, time that I would have had to spend by investigating then fixing those bugs. Time that other people would also have wasted: the customer's, the product manager's, the manual tester's, etc.

Answer 8

I can run fifteen unit tests in the time I can run one integration test.

I can get 80% code coverage with unit tests that mock the whole storage backend. (I only had to write one big mock and three individual-point mocks.)

The storage backend changes slowly and has very few regression bugs.

The middle layers have quite a few regression bugs, and would have more if I didn't have a battery of tests that I can run already.

I'm writing more unit tests. Code quality is rising. Bug regressions are falling.

Answer 9

One should really think about unit test as a development tool which help you to write something and make sure whatever you have done is correct. At least in all the ways you can think of when writing the test. These test might never fail in a CI since you made sure they pass and might never touch the code again. They maybe seem less important later on but were very important while writing the code and since you wrote the tests already might as well run them in your CI, so if anyone ever changes something it might catch an error. Also good tests often also provide example code how to use something.

For this you ofc need to be able to properly test the code. I work on a lot of code where it is hard to test individual pieces since as you also said, you have to mock half the system and probably make more mistakes in the test than in the code to be tested.

What also often happens is that a test just is there to detect changes, where the output of a unit was hard coded into the test at the time of writing. These can be helpful to some extent but it should be clear (e.g. by adding comments, documentation) that this tests only tests for changes and not for correctness. So that people not end up trying to "fix" something that was never broken just different.

The times were tests really help are if you have good define requirements or expected behaviors and you can write down what your unit should output for a given input.

Two examples form my work maybe. I once wrote run length encoding class (RLE) which takes any multi dimensional array and turns it into a run length encoding. It can convert both ways and offered functions which directly worked on the encoded data.

Now while writing this class I obviously want to try out the functions and see if there are runtime errors, if the output is what I expect and so on. So I have to write code in some way to use my new class. This is where I can just write a unit test. I have well defined functions with well defined behavior. I can come up with a sample input, come up with what output I expect from a function and test against it.

This way I was able to compare what my code does to what I do. Ofc what I did manually can be wrong and it was partly, but then I was able to identify what is the difference, why and where the errors was. I wrote a test for all the possible ways to use it I could think of and after all passed I was quite sure my code works and I also have now some nice example code how to use the class, for other developers to look at.

A second example I was wrapping a class. I inherited from a class and wanted to extend some interfaces and change some behaviors but very specific ones only. So I wrote a test and made an instance from my parent class and an instance from my new child class and just compared the outputs of all the functions which should behave exactly the same, if they are identical all is fine. Doing this I found out because of my changes some things were not really possible anymore. For some inputs I got weird behavior now because of my changes, which I would have never expected. In the end I needed to restrict some inputs in my child class and throwing a clear error if someone provided something I was not able to support. It was quite the corner case, with an input no one normally should provide but you never know who will use your code in the future and how.

Ofc finding this problem was partly due to the experience I had in writing tests. Trying to provide a lot of different inputs aiming to break something. Writing good tests is a skill, developing a mindset where you try to break something as hard as you can.

Tests can really help in being confident that the code you wrote does what is supposed to do. Without it there is always this "I think it works", with a test you can say "I know this works". Ofc the tests still an be wrong and incomplete, but now if you catch a bug, you can revisit the test and improve it so in the future it catches the bugs and maybe more similar ones, which it original missed.

As others also said, you should mainly use tests where it makes development easier where writing a test is not mocking half the system. It will help you to find bugs before code leaves your PC.

Ofc sometimes you have to write tests to reach metrics like code and branch coverage and you will probably write a lot of stupid tests which do not really help. But that doesn't mean the concept of testing is flawed and useless. You can definitely write tests which help you and future developers and make your code better.

Answer 10

Unit tests should be a by-product of development

In particular, if you're working on a large application with lots of classes, how can you test whether a new feature works? At my work, that requires compiling the whole application, loading it into a web server, starting the server, logging in as a user, navigating the menus to get to where I made a change, then testing the change.

It's a lot faster and easier to write a unit test (JUnit, because this is Java) which my IDE can compile and run much quicker. So I develop some new feature with the unit test giving me output. Often the unit test doesn't even have "asserts", it just has println statements to show me what my code is producing. Then, when I'm done, I put an assert in there (or even "assert this doesn't throw an exception"), and it becomes part of the permanent library of unit tests for the growing application.

It is true that many unit tests are unnecessary

Some of these unit tests, of course, will never fail and thus never provide meaningful information. Let's say I write a function to calculate the area of a rectangle. I could get it wrong at first because of a typographic error, but once I get it right, it'll never fail again. The test is therefore pointless. Still: is it worth the effort to judge and prune out old unit tests? What if you judge incorrectly, and something you think will never fail might actually have a failure condition?

Answer 11

and yet I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test

Of course. Unit tests won't just start failing, there's no randomness to it. Yet what happens if someone decides to improve the performance of some function or make it more readable? There may be something they change that will break the code under certain circumstances. I don't have an example handy, but that has happened before.

A big benefit I've found though is that unit tests help me write better code. If I'm having trouble writing the tests, generally I can find a better way to design the calls so that they are easier to test, and as a side effect they are more elegant and easier to call. Also writing unit tests helps me think about issues I could overlook if I just write a function to perform the happy path.

Answer 12

To be honest, I struggle with the whole concept. I'm now working on (for me) a very large and complex system with thousands of unit tests ... and yet I'm unaware of any instance of a unit test ever failing for any reason other than because the developer changed the unit under test without changing the unit test.

One of the reasons for writing unit tests is to protect you from your future self

While you may never have seen one of these tests failing (on a build pipeline for example) what you have probably not seen is how often they fail on individual developer machines as that person is refactoring.

I deal with a complex system with good unit test coverage and I have regularly refactored some code to find that my tests fail locally before I commit - telling me that I've not entirely understood what the code was doing before I changed it.

When you initially write the code, you (hopefully) have an in depth understanding of what the code is doing.

When you return to this code in six months time, this understanding will have diminished and you may no longer have a full understanding of some of the edge cases.

If you write the unit tests to validate what the intended behaviour of the code was and have covered as many edge cases as you reasonably can, then hopefully 'future you' can rely on these when making changes.

I have found using something like Bddfy to provide a structure to the unit tests in a way that describes a feature/scenario to be very useful, e.g.

[Fact]
public void CardHasBeenDisabled()
{
    this.Given(s => s.GivenTheCardIsDisabled())
        .When(s => s.WhenTheAccountHolderRequests(20))
        .Then(s => s.CardIsRetained(true), "Then the ATM should retain the card")
            .And(s => s.AndTheAtmShouldSayTheCardHasBeenRetained())
        .BDDfy(htmlReportName: "ATM");
}

And allows you to align how the unit tests are written with the language in your feature tickets.

Answer 13

I myself was skeptical of the "unit testing". And I am still somewhat skeptical about the common/usual practices around unit testing.

The first time I got the value of unit testing was, when I saw a bug report was converted to a test. The bug resolves when the test passes, easy and simple! Then I knew, all the tests there, are some kinds of checks against bugs, reported or anticipated. And as all those tests has to be passed before the software is shipped, those bugs are not coming back again. So, these tests are defining and guarding the expected behavior/outcome of the software.

Now if we go back to the first principle, what is "testing"? Not the "unit testing", just "testing"? Is it checking if a software is doing what it was supposed to do. If it is an application, QA people do the check. But that is a time-consuming, error-prone, manual job. Wouldn't it be nice if we can automate it and then run the automated tests every time a task is done, or any code is updated?

But everything cannot be automated. The software is likely dependent on database or some kind of external services. Also, the user is going to use different kind of inputs etc. So, now we can either mock the external affairs, or instrumentalize them. The first type of tests with mocking are now called unit tests, the second type is integration test.

And both of these tests are treating the software as a black box, implementation is hidden. So, these tests would not change with implementations or refactoring. But developers may want to test parts of their own implementations, which hidden from the external tests, for their own peace of mind. Then there can be many small tests alongside the implementation. But this kind of tests will likely change with implementation. I love Rust for making this type of test so easy. In any file, we can make a test block/module, which have access to the functionalities in the file and can test them. No need to change access modifiers just for testing. After using Rust's testing style, testing in other languages/frameworks feels quite artificial to me.

At the end, I am still skeptical about the current testing practices in the development world. The Java-like strict OOP, layered architecture, onion architecture, artificial interface-per-class makes the unit tests just some obligations to fulfil, without adding much real value. That is my current opinion. Thanks for asking the question!

Answer 14

Good unit tests provide the assurance that a bug is somewhere else. Unit tests differ from integration or system tests in that they can be pretty complete. An integration test can never cover the entire parameter space; a unit test can, at least with partitioning. A well-written unit test therefore is a guarantee that the unit works as expected. If you look for a bug, look elsewhere: The error must be in the overall logic or in the way your code is called. "I know that my scheduler handles up to 65535 threads with ease." "Oh."

Answer 15

Detecting breaking changes

You mention tests breaking

because the developer changed the unit under test without changing the unit test.

A unit test, both by naming/commenting and through what it actually does to test, should capture an expectation or requirement of your code. If you change code behavior and that causes a test to fail, examine the failing test and rather than trying to figure out why it fails, figure out why it tests what it does.

• Does it check that your code complies with some documented API? Things like return type, exception types, when which exceptions get thrown, etc. If your change makes this kind of test fail, then it will (or might) break customers' code or cause bugs in other places (external or internal customers, or just other components you or your team own that rely on this code behaving according to the spec). Don't change this test to accommodate the code change unless you are changing the documented behavior (and in that case, consider changing the test first, then the code, and plan to release with a new version number).
• Does it validate a constraint like performance/timeout? If the new implementation doesn't meet expectations here, you may run into SLA issues.
• Does it verify that only authenticated or authorized requests are accepted for an action?

If the solution is very clearly "update the test so it matches the function", then the old version of the test didn't really add value because it doesn't enforce anything you actually care about, and the new version most likely doesn't either.

Answer 16

Super late to the party here but I have a short answer to the question "what is the point of unit tests?" It's a very good question that I've thought about a lot over the years. Here's my answer:

Unit tests verify that code does what you (the developer) intended to it to do.

The key thing to note here is that they don't tell you that the code does what it's supposed to do. It only tells you that you have correctly implemented what you were trying to do.

There are two high-level classes of coding errors:

1. Correctly addressed requirements but made errors in the implementation.
2. Requirements were not addressed correctly.

*Note that these are not mutually exclusive. You can fail at both simultaneously. That is, you can create a flawed execution of a misunderstanding of your requirements. Regardless, it's important to distinguish between the two.

Unit tests help with #1 only. That's really the upshot. While they can seem tedious and time-consuming, if you understand that you are addressing #1 only, they can actually save you a ton of time in the long run; especially if you are working on a team. When people start using them to try to address #2, that's when they tend to become worthless at best.

Answer 17

The question could be asked broader: What are tests for (when they couldn't prevent bugs)?

First: (Unit) Tests are just code. So it applies that they also could contain errors. So it might be a case that you wrote a test and missed a corner case.

Second: (Unit) Tests slow the development down. Your codebase increases. More work has to be done. They have their own costs.

Third: You can perfectly do projects without any (Unit) Tests at all. If your customer is happy it'll make your boss happy which in turn makes you happy.

So why are we doing this?

Tests are a codfied form of expectations written down which could be repeatedly verified. So as long as your tests are "green" you know everything is as good (or bad) as expected in comparison to the last time you ran the tests. So making any meaningful change is backed up by a safety rope.

If an error occurs it might be that the code written is wrong or the test is wrong or maybe both are wrong. But once fixed you have a new set of fixed expectations for the next time.

Answer 18

Here's an example rather than a full answer.

I'd just written a Django Widget (subclass of TextInput) which used about 30 lines of Jquery to transform the field into something more complicated with radio buttons for the most frequently chosen values, but "Other" and the TextInput if none of them are applicable.

Anyway, I'd tested it manually and it was working and I almost didn't write a test. Because testing JS form manipulation requires using Selenium, which is slow and fairly complicated to write. And, it was working, and I'd clicked all the buttons in every way I could think of....

Anyway, something (instinct? experience?) nagged at me and I did write the test. And guess what? It found a bug I'd missed. One that could have been embarassing, though not damaging, when a user found it. A fairly subtle interaction between the front and back end codes.

With Python, I find that having good test coverage is thre most important thing. Ideally every line of code gets exercised by running all the tests (with a few obvious exceptions like asserting or raising "this can't happen"). I tend to write tests which other would call "too complex" which run a real-world transaction or part thereof through the system. These are the ones which are apt to fail after later "maintenance" with unforseen side-effects. It may even take a bit of further debugging to work out why they are failing. Who cares? The main thing is to know before production,. that something you changed recently, broke something else you did not expect to be affected.

(And yes, tests frequently require trivial changes because trivial and correct changes have been made to the parts of the code they test. This is un-avoidable!)

Answer 19

the developer changed the unit under test without changing the unit test. So, in essence, the unit test discovered that it wasn't in sync with the code it was testing, but that's all.

This sounds like your unit tests may be too granular.

Unit tests should be implemented at the level of units that have specific contracts; the tests confirm that they still fulfill their contracts. Code changes should only involve the implementation, not the visible behavior of the unit. So you shouldn't need to change the tests for existing units often.

As other answers suggest, this likely results from mandating thorough unit tests without thinking the implications through. Adding unit tests to a legacy application whose design itself wasn't designed rigorously may also be part of the problem -- there are no documented interfaces that serve as the obvious places for tests. So tests were simply added for every sizeable function, regardless of whether it really makes sense. And if there are too many tests, updating them will be a chore that often gets overlooked.