When is it appropriate to not unit test?

Question

I work in a small company as a solo developer. I'm the only developer at the company in fact. I have several (relatively) large projects I've written and maintain regularly, and none of them have tests to support them. As I begin new projects I often wonder if I should try a TDD approach. It sounds like a good idea, but I honestly can never justify the extra work involved.

I work hard to be forward-thinking in my design. I realize that certainly one day another developer will have to maintain my code, or at least troubleshoot it. I keep things as simple as possible and I comment and document things that would be difficult to grasp. And the fact is these projects aren't so big or complicated that a decent developer would struggle to comprehend them.

A lot of the examples I've seen of tests get down to the minutiae, covering all facets of the code. Since I'm the only developer and I'm very close to the code in the entire project, it is much more efficient to follow a write-then-manually-test pattern. I also find requirements and features change frequently enough that maintaining tests would add a considerable amount of drag on a project. Time that could otherwise be spent solving the business needs.

So I end up with the same conclusion each time. The return on investment is too low.

I have occasionally setup a few tests to ensure I've written an algorithm correctly, like calculating the number of years someone has been at the company based on their hire date. But from a code-coverage standpoint I've covered about 1% of my code.

In my situation, would you still find a way to make unit testing a regular practice, or am I justified in avoiding that overhead?

UPDATE: A few things about my situation that I left out: My projects are all web applications. To cover all my code, I'd have to use automated UI tests, and that is an area where I still don't see a great benefit over manual testing.

Answer 1

A lot of the examples I've seen of tests get down to the minutiae, covering all facets of the code.

So? You don't have to test everything. Just the relevant things.

Since I'm the only developer and I'm very close to the code in the entire project, it is much more efficient to follow a write-then-manually-test pattern.

That's actually false. It's not more efficient. It's really just a habit.

What other solo developers do is write a sketch or outline, write the test cases and then fill in the outline with final code.

That's very, very efficient.

I also find requirements and features change frequently enough that maintaining tests would add a considerable amount of drag on a project.

That's false, also. The tests are not the drag. The requirements changes are the drag.

You have to fix the tests to reflect the requirements. Whether their minutiae, or high-level; written first or written last.

The code's not done until the tests pass. That's the one universal truth of software.

You can have a limited "here it is" acceptance test.

Or you can have some unit tests.

Or you can have both.

But no matter what you do, there's always a test to demonstrate that the software works.

I'd suggest that a little bit of formality and nice unit test tool suite makes that test a lot more useful.

Answer 2

Imagine that you had a suite of tests that could run in an eyeblink and would light up a green or red light. Imagine that this suite of tests tested everything! Imagine that all you had to do to run the suite of tests was to type ^T. What power would this give you?

Could you make a change to the code without fear of breaking something? Could you add a new feature without fear of breaking an old feature? Could you clean up messy code quickly without fear of doing damage?

Yes, you could do all those things! And what would happen to your code over time? It would get cleaner and cleaner because there would be no risk to cleaning it.

Let's imagine that you had a little fairy on your shoulder. Every time you wrote a line of code, the fairy would add something to the test suite that tested that that line of code did what it was intended to do. So every couple of seconds you could hit ^T and see that the last line of code you wrote worked.

How much debugging do you think you would do?

If this sounds like fantasy, you're right. But the reality is not much different. Replace the eyeblink with a few seconds, and the fairy with the TDD discipline, and you've pretty much got it.

Let's say you are coming back to a system you built a year ago, and you've forgotten how to create one of the central objects. There are tests that create that object every way it can be created. You can read those tests and jog your memory. Need to call an API? There are tests that call that API every way it can be called. These tests are little documents, written in a language you understand. They are completely unambiguous. They are so formal that they execute. And they cannot get out of sync with the application!

Not worth the investment? You've got to be kidding me! How could anyone NOT want that suite of tests? Do yourself a favor and stop quibbling over silliness. Learn to do TDD well, and watch how much faster you go, and how much cleaner your code is.

Answer 3

The mistake you are making is that you are seeing testing as a time investment with no immediate return. It doesn't necessarily work like that.

Firstly writing tests really focusses you on what this part of your code needs to do.

Secondly running them reveals bugs that would otherwise come up in testing.

Thirdly running them sometimes shows up bugs that wouldn't otherwise come up in testing and then would really bite you in the ass in production.

Fourthly if you hit a bug with a system that is running and create a unit test for it, you will not be able to re-introduce that bug later. That can be a really big help. Reintroduced bugs are common and very annoying.

Fifthly if you ever need to hand code over to someone else, a test suite will make their life far easier. Also if you have ignored a project and come back to it after a few years, you won't be so close to it any more and it will be helpful to you as well.

My experience has consistently been that across the development of a project, having decent unit tests has always made the process quicker and more reliable.

Answer 4

The guys at JUnit (Java Unit test framework) have a philosophy that if it is too simple to test, don't test it. I highly recommend reading their Best Practices FAQ, as it is fairly pragmatic.

TDD is a different process of writing your software. The basic premise behind unit testing is that you will spend less time in the debugger stepping through code, and more quickly figure out if your code change accidentally breaks something else in the system. That fits in with TDD. The TDD cycle is like this:

1. Write a test
2. Watch it fail (prove you have something to do)
3. Write just what is needed to make the test pass--no more.
4. Watch it pass (yay!)
5. Refactor (make it better)
6. Wash, rinse, and repeat

What is less obvious about applying TDD is that it changes the way your write code. By forcing yourself to think about how to test/validate that the code is working, you are writing testable code. And since we are talking unit testing, that usually means that your code becomes more modular. To me, modular and testable code is a big win up front.

Now, do you need to test things like C# properties? Imagine a property defined like this:

bool IsWorthTesting {get; set;}

The answer would be "no" it's not worth testing, because at this point you are testing the language feature. Just trust that the C# platform guys got it right. Besides, if it failed, what could you do to fix it?

Also, you will find that there are certain parts of your code that very well will be too much effort to test properly. That means don't do it, but make sure you test the code that uses/is used by the tricky problem:

• Checked exceptions that can only occur if an install went bad. Java has a ton of these. You are required to write a catch block or declare the checked exception even if there is no way it can fail without hacking the installed files.
• User interfaces. Finding the control under test, and invoking the right events to simulate a user's actions are very troublesome, and in some cases impossible. However, if you use the Model/View/Controller pattern, you can make sure your model and controllers are tested and leave the view part to manual testing.
• Client/server interactions. This is no longer a unit test, and is now an integration test. Write all the parts that go up to sending and receiving messages over the wire, but don't actually go over the wire. A good approach is to reduce the responsibility of the code that actually talks over the wire to the raw communications. In your unit test code, mock communication object out to make sure the services are behaving as you expect.

Believe it or not, TDD will help you fall into a sustainable pace of development. It's not because of magic, but rather because you have a tight feedback loop and you are able to catch really dumb mistakes quickly. The cost of fixing those mistakes is essentially constant (at least enough for planning purposes) because the small mistakes never grow up to be big mistakes. Compare that with the bursty nature of code binge/debug purge sprints.

Answer 5

Testing is gambling.

Creating a test is a bet that the cost of bugs in a unit occurring and not catching them with that test (now, and during all future code revisions) is greater than the cost of developing the test. Those test development costs include things like payroll for added test engineering, added time-to-market, lost opportunity costs from not coding other stuff, and etc.

Like any bet, sometimes you win, sometimes you lose.

Sometime late software with far fewer bugs wins out over quick but buggy stuff that gets to market first. Sometimes the opposite. You have to look at the statistics in your particular field, and how much management wants to gamble.

Some types of bugs might be so unlikely to be made, or to make it out of any early sanity testing, as to be statistically not worth the time to create additional specific tests. But sometimes the cost of a bug is so great (medical, nuclear, etc.) that a company must take a losing bet ( similar to buying insurance). Many apps do not have such a high failure cost, and thus don't need the higher uneconomical insurance coverage. Others do.

Answer 6

You have to balance the cost of testing with the cost of bugs.

Writing a 10 line unit test for a function that opens a file, where the failure is "file not found" is pointless.

A function that does something complex to a complex data structure - then obviously yes.

The tricky bit is in-between. But remember the real value of unit tests isn't testing the particular function, it's testing the tricky interactions between them. So a unit test that spots that a change in one bit of code, breaks some function in a different module a 1000 lines away, is worth its weight in coffee.

Answer 7

My advice is to only test the code that you want to work properly.

Don't test the code that you want to be buggy and to cause problems for you down the road.

Answer 8

I often wonder if I should try a TDD approach. It sounds like a good idea, but I honestly can never justify the extra work involved.

TDD and Unit Testing are not the same thing.

You can write code, then add unit tests later. That is not TDD, and is a lot of extra work.

TDD is the practice of coding in a loop of Red Light. Green Light. Refactor iterations.

This means writing tests for code that does not yet exist, watching the tests fail, fixing the code to make the tests work, then making the code "right". This often saves you work

One of the advantages to TDD is that it reduces the need to think about trivia. Things like off-by-one errors disappear. You don't have to go hunting through API documentation to find out if the list it returns starts at 0 or 1, just do it.

Answer 9

I worked on a system where we tested almost everything. The notable executions to testing were the PDF and XLS output code.

Why? We were able to test the parts that gathered the data and built the model that was used to create the output. We were also able to test the parts that figured out what parts of the model would go to the PDF files. We weren't able to test if the PDF looked ok because that was totally subjective. We weren't able to test that all the parts in a PDF were readable to a typical user because that was also subjective. Or if the choice between bar and pie charts was correct for the dataset.

If the output is going to be subjective, there is little unit testing you can do what is worth the effort.

Answer 10

For many things, a 'write-then-manually-test' takes no more time than writing a couple tests. The time savings come from being able to re-run those tests at any time.

Think of it: If you have some decent feature coverage with your tests (not to be confused with code coverage), and let's say you have 10 features - clicking a button means you have roughly, 10 yous re-doing your tests... while you sit back and sip your coffee.

You also don't have to test the minutae. You can write integration tests that cover your features if you don't want to get down to the nitty gritty details... IMO, some unit tests get too fine-grained testing the language and platform, and not the code.

TL;DR It's really never appropriate because the benefits are just too good.

Answer 11

Professional developers write unit tests because, in the longer term, they save time. You are going to test your code sooner or later, and if you don't your users will, and if you have to fix bugs later they are going to be harder to fix and have more knock on effects.

If you are writing code with no tests and have no bugs then fine. I don't believe you can write a non-trivial system with zero bugs though, so I assume you are testing it one way or another.

Unit tests are also crucial to prevent regressions when you modify or refactor older code. They do not prove your change hasn't broken old code but they give you a lot of confidence (so long as they pass of course:) )

I would not go back and write a whole batch of tests for code that you have already shipped, but next time you need to modify a feature I'd suggest trying to write tests for that module or class, get your coverage up to 70%+ before you apply any changes. See if it helps you.

If you try it and can honestly say it was no help then fair enough, but I think there is enough industry evidence that they help to make it at least worth your while trialling the approach.

Answer 12

It seems like most answers are pro-TDD, even though the question wasn't asking about TDD but about unit tests in general.

There isn't a completely objective rule behind what to unit test or not to unit test. But there are a couple of times where it seems many programmers do not unit test:

1. Private methods

Depending on your OOP philosophy, you might create private methods to decouple complex routines from your public methods. Public methods are usually intended to be called in many different places and used often, and the private methods are only really called by one or two public methods in a class or module in order to something very specific. It is usually sufficient to write unit tests for public methods but not the underlying private methods that make some of the magic happen. If something goes wrong with a private method, your public method unit tests should be good enough to detect these issues.

2. Things you already know should work(or things tested by someone else)

A lot of new programmers go against this when they are first learning to test, and think that they need to test every single line being executed. If you are using an external library and its functionality is well tested and documented by its authors, it's usually pointless to tests the specific functionality in unit tests. For example, someone might write a test to make sure that their ActiveRecord model persists the correct value for an attribute with a "before_save" callback to the database, even though that behavior itself is already thoroughly-tested in Rails. The method(s) that the callback is calling, perhaps, but not the callback behavior itself. Any underlying problems with imported libraries would better be revealed through acceptance tests, rather than unit tests.

Both of those could apply whether you're doing TDD or not.

Answer 13

Two very good answers I have come across are here:

1. When to unit-test vs manual test
2. What not to test when it comes to Unit Testing?

The justifications for avoiding perceived overhead:

• Immediate Time/Cost saving for your company
• Potential Time/cost saving in troubleshooting/maintainability/extension in the long run even after you're gone.

Would you not want to leave a great product from your side as the proof of quality of your work? Speaking in selfish terms, is it not better for you that you do?

Answer 14

Ken, I and many many other developers out there have arrived at the same conclusion as you several times throughout our careers.

The truth that I believe you'll find (as have many others) is that the initial investment of writing tests for your application may seem daunting, but if written well and targetted at the correct parts of your code, they can really save a ton of time.

My big problem was with the testing frameworks available. I never really felt like they were what I was looking for, so I just rolled my own very simple solution. It really helped to bring me around to the "dark side" of regression testing. I'll share a basic pseudo snippet of what I did here, and hopefully you can find a solution that works for you.

public interface ITest {
    public string Name {
        get;
    }
    public string Description {
        get;
    }
    public List<ITest> SubTests {
        get;
    }
    public TestResult Execute();
}

public class TestResult {
    public bool Succesful {
        get;
        set;
    }

    public string ResultMessage {
        get;
        set;
    }

    private Dictionary<ITest, TestResult> subTestResults = new Dictionary<ITest, TestResult>();
    public Dictionary<ITest, TestResult> SubTestResults {
        get {
            return subTestResults;
        }
        set {
            subTestResults = value;
        }
    }
}

The only tricky part after that is figuring out what granularity level you think is the best "bang for the buck" for whatever project you're doing.

Building an address book will obviuosly require way less testing than an enterprise search engine, but the fundamentals don't really change.

Good luck!

Answer 15

You shall skip unit testing only when doing so will be very costly. In the rest of situations it is very worth it.

In the end you have to test your code somehow. Unit testing is just keeping that test around.

That allows two important things. One being able to refactor the code, and being sure that side effects don't propagate. Which is so usual, and hard to debug.

The second is being more exhaustive with the cases you test. When you test without a unit test you tend to forget about corner cases, like null or malformed input.