When is unit testing inappropriate or unnecessary? [duplicate]

Question

It seems to be generally assumed (on Stack Overflow at least) that there should always be unit tests, and they should be kept up to date. But I suspect the programmers making these assertions work on different kinds of projects from me - they work on high quality, long-lasting, shipped software where bugs are A Really Bad Thing.

If that's the end of the spectrum where unit testing is most valuable, then what's the other end? Where are unit tests least valuable? In what situations would you not bother? Where would the effort of maintaining tests not be worth the cost?

Answer 1

I will personally not write unit tests for situations where:

1. The code has no branches is trivial. A getter that returns 0 doesn't need to be tested, and changes will be covered by tests for its consumers.

2. The code simply passes through into a stable API. I'll assume that the standard library works properly.

3. The code needs to interact with other deployed systems; then an integration test is called for.

4. If the test of success/fail is something that is so difficult to quantify as to not be reliably measurable, such as steganography being unnoticeable to humans.

5. If the test itself is an order of magnitude more difficult to write than the code.

6. If the code is throw-away or placeholder code. If there's any doubt, test.

Answer 2

Unless you are going to write code without testing it, you are always going to incur the cost of testing.

The difference between having unit tests and not having them is the difference between the cost of writing the test and the cost of running it compared to the cost of testing by hand.

If the cost of writing a unit test is 2 minutes and the cost of running the unit test is practically 0, but the cost of manually testing the code is 1 minute, then you break even when you have run the test twice.

---

For many years I was under the misapprehension that I didn't have enough time to write unit tests for my code. When I did write tests, they were bloated, heavy things which only encouraged me to think that I should only ever write unit tests when I knew they were needed.

Recently I've been encouraged to use Test Driven Development and I found it to be a complete revelation. I'm now firmly convinced that I don't have the time not to write unit-tests.

In my experience, by developing with testing in mind you end up with cleaner interfaces, more focussed classes & modules and generally more SOLID, testable code.

Every time I work with legacy code which doesn't have unit tests and I have to manually test something, I keep thinking "this would be so much quicker if this code already had unit tests". Every time I have to try and add unit test functionality to code with high coupling, I keep thinking "this would be so much easier if it had been written in a de-coupled way".

Comparing and contrasting the two experimental stations that I support. One has been around for a while and has a great deal of legacy code, while the other is relatively new.

When adding functionality to the old lab, it is often a case of getting down to the lab and spending many hours working through the implications of the functionality they need and how I can add that functionality without affecting any of the other functionality. The code is simply not set up to allow off-line testing, so pretty much everything has to be developed on-line. If I did try to develop off-line then I would end up with more mock objects than would be reasonable.

In the newer lab, I can usually add functionality by developing it off-line at my desk, mocking out only those things which are immediately required, and then only spending a short time in the lab, ironing out any remaining problems not picked up off-line.

---

TL;DR version:

Write a test when the cost of writing the test, plus the cost of running it as many times as you need to is likely to be less than the cost of manually testing it as many times as you need to.

Remember though that if you use TDD, the cost of writing tests is likely to come down as you get better at it, and unless the code is absolutely trivial, you probably end up running your tests more often than you expect.

Answer 3

Personally, I generally do not write unit tests for:

• Changes to existing non-unit-tested (legacy) code. While theoretically it's never too late to start unit testing, in practice the gains from adding unit tests to code that was never TDDed are small. The sole advantage is regression prevention; you can see what effect a change makes on areas you may or may not have expected it to affect. However, the unit test only proves that the code still works the way it did when you wrote the test; because the test wasn't written based on the original requirements, the test does not prove the current implementation is "correct". Making unit-tested changes to such a codebase is even more of a nightmare in my experience then simply making the change and manually exercising affected areas of the program; you first have to write a proper unit test for the code in question (which may require refactoring the code to allow for mocking and other TDD-friendly architecture), prove it passes AND that the refactor didn't break anything (and when this is the first unit test ever written for a million-LoC codebase, that can be difficult), then change the test to expect something different and change the code to make the changed test pass.

• UI Layout and top-level behavior. In some cases it's just not feasible; many IDEs with "designers" declare UI elements as private, and they shouldn't be public conceptually. Making them internal would require putting the unit tests into the same assemblies as production code (bad) and making them protected would require a "test proxy" that exposes what we need to know about the controls (also generally not the way it's done). Even if it was acceptable to make all controls public and thus have access to their type, position, formatting properties, etc, tests that verify the UI is designed the way you expect are extremely brittle. You can usually still unit-test event handlers in codebehinds/viewmodels and then it's trivial to prove that clicking the button fires the handler.

• Database Persistence. This is, by definition, an integration test. You always mock the repository when dealing with code that passes data to the repository, and you mock the ORM's hooks when developing within the repository itself. While there are "database unit tests" and you can "mock" a database using something like SQLite, I find that these tests just never fit at the unit level.

• Third-party code. You test YOUR code however you want, and trust that the developers of the third-party tools you're using (including components of the .NET Framework itself like WPF, WCF, EF, etc) did their job right, until proven wrong. A unit test that proves you make the correct WCF call does not require actually crossing the service proxy, and it is unnecessary to test that a particular WCF service call works correctly in a unit test (whether it works or not depends on environment-specific variables that may be perfect in a local testing environment but not at the build-bot, or in the QA, Staging or UAT environments; this is where higher-level integration, AAT and manual testing still have value).

In almost every other case, a test-first methodology has significant value in enforcing requirements and YAGNI, preventing regression, identifying requirements conflicts, etc. Even in cases of "trivial" code, that verification that "yes, this property does still return this value and so you can trust that in other tests" has value to me as a developer.

Answer 4

There is a trade-off between the cost of writing unit tests, and the value they add. Neglecting this risks the possibility of over-testing.

blueberryfields touched on one variable that affects the value of unit tests: stability of the project (or pace of development). I'd like to expand on that.

The value to having unit tests roughly follows the following plot (yay, MS Paint!):

1. In the first zone, the code is unstable to the point that unit tests will often need to be rewritten or entirely scrapped. This might be a prototype that you know won't be used. Or it might be a project where you are new to the programming language or framework (and are often needing to redo large amounts of code).

   Unit tests are good for refactoring at the module level (i.e. changing functions). The problem is when code is unstable at the architectural level, where entire modules might be rewritten, removed, or completely redesigned. The unit tests then must be changed/removed/added, which is a burden.

   At this stage, limiting to system level integration tests, or even manual testing is a better approach (which one you choose again being dependent on the cost of writing the test vs. the value you get).

2. The unit tests add the most value in the second zone, where code becomes stable above the module level, but not yet at the module level. If you have a stable codebase, and you are adding a module that has a high chance of staying, this is when unit testing is a big win.

3. Unit tests also begin to have less value when the stability of the code under test is very high. Testing a simple wrapper of a standard library function is one example where unit testing is overkill.

   If a piece of code you are writing will likely never need to change, and you are confident you've got it right (maybe it's super simple?), there is less value to testing.

Unit testing becomes beneficial when the utility is greater than the cost, as shown in the below graph in zone II.

This cost line could be at different levels for different people (line 1 vs line 2). If you are new to the test framework used in a project, this increases the level, and makes the net utility less. A higher cost (line 2) shrinks the range that unit testing is appropriate.

The cost line doesn't necessarily need to be flat. Especially when you consider other variables such as project size, and whether a test framework already exists for the project, this might be so.

Answer 5

Where are unit tests least valuable?

You should be writing unit tests only for logical code. Definition of logical code is below:

Logical code is any piece of code that has some sort of logic in it, small as it may be. It’s logical code if it has one or more of the following: an IF statement, a loop, switch or case statements, calculations, or any other type of decision-making code.

If that's the end of the spectrum where unit testing is most valuable, then what's the other end?

• Tightly coupled code can't be easily tested. Sometimes it can't be tested at all. For example ASP.NET web forms are a nightmare and sometimes impossible to test, as opposed to ASP.NET MVC which is built with unit testing in mind.

• I may choose not to write tests for very small and simple projects. This is perfectly justifiable in a world with limited time and resources. Yes, it's risky, but it's perfectly acceptable when the company you work for has a limited time to market.

What situations would you not bother? Where would the effort of maintaining tests not be worth the cost?

Once unit tests are in place, their maintenance cost becomes very insignificant. Unit tests with high code coverage give you a valuable sense of confidence that system behaves how you expect it to. The same applies to integration tests, but that's not your question.

My answer is based on the following assumptions:

• You know difference between unit, integration and functional tests.
• You are proficient with one of the unit-testing frameworks, and it takes you very little effort to write a unit test.
• You don't feel like you are doing extra work when you are writing unit tests.

Reference: Art of Unit Testing

Answer 6

Kent Beck (the man behind the idea of Unit Testing) says that it's fine not to write Unit Tests when working on an idea (eg. prototype) that you might throw away (if you decide not to work further on). On the contrary, if you write your prototype and decide to go for it, Unit Testing will be helpful for all the reasons covered in the previous answers.

Answer 7

You should really have tests for any code that you intend to run on more than 1 occasion.

If you intend to run it once then throw it away, tests may not be useful, though if the domain is complex, I might well write the tests anyway to help me with my design and implementation.

Otherwise, even if the only thing the tests do is confirm that your assumptions are still correct, they are valuable. And any code that lasts more than a single session will need modifications, at which point the tests will pay for themselves 10 times over.

Answer 8

I have no clue about unnecessary but I'll shoot for when I think Unit Tests are inappropriate in that they add cost without proportional benefit:

If our organization / dev team isn't doing any unit tests, then unless you are able to convince your fellow developers of running, fixing (and also creating on their own) unit tests, the tests you create aren't going to be run and maintained. (This implies you are working on a piece of software together with someone else.)

Not everyone is doing unit testing, they're sometimes shoddily supported by local team culture, and in such environments you first need to get them in place for the majority of devs and probably also for the build-process ("[continuous] integration") before actually starting to write unit tests makes sense.

Given time pressure and delivery dates and "your" relative standing with the team -- the time remaining until writing unit tests makes sense can easily be months or years.

Answer 9

Sometimes you don't have time to unit test. That's okay.

External constraints exists and you may not feel the need for tests in a given moment, as you may think you have a firm grasp on the whole system state and know how changes will ripple trough it.

But I still believe you should feel a little guilty about it. So when things start to fall apart, you will know it's because: "oh, yeah, I'm not testing stuff, let's start testing."

Before that moment, the feel of guilt will help you avoid certain architectural choices that would prevent you from being able to automate code tests later.

Answer 10

The cost isn't in maintaining tests, but rather in not maintaining working code. If you divorce testing and implementation, you are assuming that what you write will work without the benefit of proof. Tests give you that proof, and the confidence to maintain the code as time passes. Sure you might think that you'll never revisit that app you thought was a throw-away, however the reality is that any time you spend writing software is an investment that your employer doesn't want to treat lightly. Let's face it, software developers don't generally come cheap, and time spent on code to throw away is money simply burned to your employer.

If you are spiking a problem, writing a few lines of code to see how something works, or simply throwing together an example for Programmers.SE, then it's really a case of creating something that you fully intend to throw away once you've satisfied your understanding of a problem. In such cases, unit testing can be a hindrance to the task and can safely be left out. In all other cases however, I'd say that testing is always appropriate, even if the program seems relatively unimportant, and you should always aim to write software to a professional standard with enough test support to reduce the difficulty in maintaining the code in the future.

Answer 11

The amount and quality of unit testing to be done in a project is directly proportional to the pace of development and the required stability of the product.

At one end, for a one-off script, trivial demo, or in a quick-paced-startup environment during the very early stages, it's unlikely unit tests will matter or be useful for anything whatsoever. They are almost always, almost certainly, a waste of time and resources.

At the other end, when writing software to control pace makers, or pilot rocket-ships, or when working at maintaining stable, highly complex, software systems (say, a compiler with heavy adoption across the multiverse) multiple layers of unit test, and other coverage, are crucial.

[There are loads of other situation specific factors which each contribute one way or the other, such as the size of your team, your budget, the talent of your developers, your preferred software dev strategy, etc... Generally, and at the highest level of abstraction, I find these two make the largest difference]

Answer 12

In my view, product/code prototypes don't require testing as they are often delivered using a completely different process and for completely different purposes than a finished product.

Imagine this common scenario. You have some sort of technology that your company is famous for, your sales team or a chief exec goes to a client and is able to sell them a different product that is leveraging your technology. They sell it there and then and promise the client they'll see a beta version in a month time. You don't have anything, even a development team.

What happens next is you quickly design a solution based on the points agreed/sold to the client, assemble a team of (often star) engineers and start writing code. It would be ideal of course if your dates didn't slip (they always do) and you managed to write code in three weeks time rather than four. In this case you have an extra week, and if you were able to secure a QA team's time for this week - you will test it. In reality you won't have an extra week and/or your QA team is booked for another two months supporting other product deliveries. Moreover you won't have any design specs, so QA team won't have any idea what they are supposed to be testing. So you end up delivering what you wrote with a bit of your own smoke testing, which isn't really a testing as such, simply because you aren't qualified to test your own code. The biggest factor here, is that the client will see some of the promised functionality very quickly. If your prototype does the job, then you're back to the drawing board, where you design and plan the actual version 1.0 - properly this time.

As far as unit-tests go, they are good to maintain certain contracts (I promise my code will do this, if it doesn't - it's broken). In case of prototyping, you may not care so much for contracts, because often enough your prototypes will not live to become version 1.0, whether because the client has went away, or because you ended up developing your version 1.0 using different technologies/tools. An example here would be a quick prototype website written very quickly on PHP in a month's time by one person, and then you assemble a team to rewrite it in JEE. Even the database will often not survive this change.

Answer 13

You should write unit tests when when the logical comprehension required to maintain or extend or communicate the logic of the application without flaws exceeds (or will likely in the future exceed) the ability of the least knowledgeable member of your team to meet that understanding.

Basically depends how much you can keep in your head at once. On your own you might have more confidence that you can code (in full knowledge of the impact of your changes) until a certain complexity has been reached. Until that point unit tests are a distraction, a hindrance and will slow you down. In a team, the threshold is going to be much lower so you pretty much need unit tests straight away to counteract group-stupidity. If you pick up a testless project from someone else (or you've forgotten about a project from a while ago) you'll want to write unit tests before you change anything because your understanding will be zero.

Note: writing less code and practising YAGNI, keeps down complexity and hence the number of required unit tests.

Answer 14

Unit tests are nice and all, but in complicated systems with lots of "moving parts", they really don't actually test much.

As we know, Unit tests ensure that given input X for a unit, you get output Y where X is all appropriate input and Y is all appropriate output. However, I find that most code does what the developer thought it should.. but not necessary what it actually should do.
(You might think of it like a gun. A gun has a bunch of precision parts manufactured separately and then assembled. It doesn't matter if each particular part is tests well for functionality if the gun explodes when actually fired. Disclaimer: I know nothing about fire arms (: )

In this cases, we need actual system tests in a model production environment. Since functionality is tested here anyway, the value of unit tests is slightly diminished.

However, unit tests are still valuable in this case because they provide assurance that functionality that you wished to remain unchanged does in fact remain unchanged. This is quite valuable because it gives you confidence for areas of the system need focused testing.

Answer 15

IMO unit tests are inappropriate (not unnecessary!) in any codebase where it would require tons of hours of refactoring or even an outright rewrite in order to introduce unit testing, because it would be a very hard sell to management to allow for that time allotment.

Answer 16

In a fast paced smaller organization, I find that developers are so pressured to stay productive and move fast that they often simply don't bother to test their own changes.

We have a general policy that states that if you make a change (whether it is a bug fix or enhancement) the developer should test it him/herself before submitting the code for further QA.

We don't actually write unit tests for most instances because our screens are mostly designed in a way where if a change is made on the screen, the rest of the logic has to be traversed through in order to find out if the change was successful regardless.

I find it imperative that the developer does their own testing, whether it is documented or not, because they are the ones who best know where the potential weak points in the logic may be.

So I would say, even if you don't write unit tests for every little part of the system, common sense testing should be done regardless.