237

I find tests a lot trickier and harder to write than the actual code they are testing. It's not unusual for me to spend more time writing the test than the code it is testing.

Is that normal or am I doing something wrong?

The questions “Is unit testing or test-driven development worthwhile?”, “We are spending more time implementing functional test than implementing the system itself, is this normal?” and their answers are more about whether testing is worth it (as in "should we skip writing tests altogether?"). While I'm convinced tests are important, I'm wondering if my spending more time on tests than actual code is normal or if it's only me.

Judging by the number of views, answers and upvotes my question received, I can only assume its a legitimate concern that isn't addressed in any other question on the website.

17
  • 20
    Anecdotal, but I find I spend roughly as much time writing tests as writing code when I TDD. It's when I slip up and write tests after the fact that I spend more time on the tests than the code.
    – RubberDuck
    Oct 14, 2015 at 9:49
  • 10
    You also spend more time reading than writing your code. Oct 14, 2015 at 12:44
  • 27
    Also, tests are actual code. You just don't ship that part to customers. Oct 14, 2015 at 12:44
  • 5
    Ideally you'll spend more time running than writing your code, too. (Otherwise, you'd just do the task by hand.) Oct 14, 2015 at 13:47
  • 7
    @RubberDuck: Opposite experience here. Sometimes when I write tests after the fact the code and the design are already pretty tidy so I do not need to rewrite the code and the tests too much. So it takes less time to write the tests. It is not a rule, but it happens to me quite often.
    – Giorgio
    Oct 14, 2015 at 17:40

9 Answers 9

216

I remember from a software engineering course, that one spends ~10% of development time writing new code, and the other 90% is debugging, testing, and documentation.

Since unit-tests capture the debugging, and testing effort into (potentially automate-able) code, it would make sense that more effort goes into them; the actual time taken shouldn't be much more than the debugging and testing one would do without writing the tests.

Finally tests should also double as documentation! One should write unit-tests in the way the code is intended to be used; i.e. the tests (and usage) should be simple, put the complicated stuff in the implementation.

If your tests are hard to write then, the code they test is probably hard to use!

6
  • 5
    It might be a good time to look into why the code is so hard to test :) Try to "unroll" complex multi-function lines that are not strictly necessary, like massively nested binary/ternary operators... I really hate unnecessary binary/ternary operator that also has a binary/ternary operator as one of the paths...
    – Nelson
    Oct 14, 2015 at 1:46
  • 64
    I beg to disagree with the last part. If you're aiming for very high unit test coverage, you need to cover use cases that are rare and sometimes outright against the intended use of your code. Writing tests for those corner cases may just be the most time consuming part of the whole task.
    – otto
    Oct 14, 2015 at 8:05
  • 2
    I've said this elsewhere, but unit testing tends to run far longer, because most code tends to follow a sort of "Pareto Principle" pattern: you can cover about 80% of your logic with about 20% of the code it takes to cover 100% of your logic (i.e. covering all edge cases takes about five times as much unit testing code). Of course, depending on the framework, you can initialize the environment for multiple tests, reducing the overall code needed, but even that takes additional planning. Approaching 100% confidence requires far more time than simply testing main paths.
    – phyrfox
    Oct 14, 2015 at 20:13
  • 3
    @phyrfox I think that's too cautious, it's more like "the other 99% of the code is edge cases". Which means the other 99% of the tests are for those edge cases.
    – Móż
    Oct 14, 2015 at 20:57
  • @Nelson I agree nested ternary operators are hard to read, but I don't think they make testing particularly difficult (a good coverage tool will tell you if you've missed one of the possible combinations). IMO, software is hard to test when it's too tightly coupled, or depends on hardwired data or data not passed as a parameter (e.g. when a condition depends on the current time, and this is not passed as a parameter). This is not directly related to how "readable" the code is, though of course, all other things equal, readable code is better!
    – Andres F.
    Oct 15, 2015 at 15:49
99

It is.

Even if you take just unit testing, it is not unusual to have more code within the tests than the actually tested code. There is nothing wrong with it.

Consider a simple code:

public void SayHello(string personName)
{
    if (personName == null) throw new NullArgumentException("personName");

    Console.WriteLine("Hello, {0}!", personName);
}

What would be the tests? There are at least four simple cases to test here:

  1. The person name is null. Is exception actually thrown? That's at least three lines of test code to write.

  2. The person name is "Jeff". Do we get "Hello, Jeff!" in response? That's four lines of test code.

  3. The person name is an empty string. What output do we expect? What is the actual output? Side question: does it match the functional requirements? That means another four lines of code for the unit test.

  4. The person name is short enough for a string, but too long to be combined with "Hello, " and the exclamation point. What happens?¹

This requires a lot of testing code. Moreover, the most elementary pieces of code often require setup code which initializes the objects needed for the code under test, which also often leads to writing stubs and mocks, etc.

If the ratio is very big, in which case you may check a few things:

  • Is there code duplication across the tests? The fact that it's test code doesn't mean that the code should be duplicated (copy-pasted) between similar tests: such duplication will make the maintenance of those tests difficult.

  • Are there redundant tests? As a rule of thumb, if you remove a unit test, the branch coverage should decrease. If it doesn't, it may indicate that the test is not needed, since the paths are already covered by other tests.

  • Are you testing only the code you should test? You are not expected to test the underlying framework of the third-party libraries, but exclusively the code of the project itself.

With smoke tests, system and integration tests, functional and acceptance tests and stress and load tests, you add even more test code, so having four or five LOC of tests for every LOC of actual code is not something you should be worried about.

A note about TDD

If you're concerned about the time it takes to test your code, it might be that you are doing it wrong, that is code first, tests later. In this case, TDD may help by encouraging you to work in iterations of 15-45 seconds, switching between code and tests. According to the proponents of TDD, it speeds up the development process by reducing both the number of tests you need to do and, more importantly, the quantity of business code to write and especially rewrite for testing.


¹ Let n be the maximum length of a string. We can call SayHello and pass by reference a string of length n - 1 which should work just fine. Now, at Console.WriteLine step, the formatting should end up with a string of length n + 8, which will result in an exception. Possibly, due to the memory limits, even a string containing n / 2 characters will lead to an exception. The question one should ask is whether this fourth test is a unit test (it looks like one, but may have a much higher impact in terms of resources compared to average unit tests) and if it tests the actual code or the underlying framework.

9
  • 5
    Don't forget a person can have the name null too. stackoverflow.com/questions/4456438/…
    – psatek
    Oct 14, 2015 at 14:17
  • 1
    @JacobRaihle I assume @MainMa means the value of personName fits in a string, but the value of personName plus the concatenated values overflow string.
    – woz
    Oct 14, 2015 at 17:16
  • @JacobRaihle: I edited my answer to explain this point. See the footnote. Oct 14, 2015 at 17:22
  • 4
    As a rule of thumb, if you remove a unit test, the branch coverage should decrease. If I write all the four tests you mentioned above, and then remove the 3rd test, will the coverage decrease ?
    – Vivek
    Oct 14, 2015 at 17:28
  • 3
    "long enough" → "too long" (in point 4)? Oct 14, 2015 at 19:04
65

I think it is important to distinguish between two types of test strategies: unit testing and integration/acceptance testing.

Although unit testing is necessary in some cases, it is frequently hopelessly over-done. This is exacerbated by meaningless metrics forced on developers, like "100% coverage". http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf provides a compelling argument for this. Consider the following issues with aggressive unit testing:

  • An abundance of pointless tests that does not document a business value, but simply exist to get closer to that 100% coverage. Where I work, we have to write unit tests for factories that does nothing else but create new instances of classes. It adds no value. Or those lengthy .equals() methods generated by Eclipse - there is no need to test these.
  • In order to make testing easier, developers would sub-divide complex algorithms into smaller testable units. Sounds like a win, right? Not if you need to have 12 classes open to follow a common code path. In these cases, unit testing can actually decrease code readability. Another problem with this is that if you chop your code into too small pieces, you end up with a magnitude of classes (or pieces of code) that seem to have no justification beyond being a sub-set of another piece of code.
  • Refactoring of highly coveraged code can be difficult, as you also need to maintain the abundance of unit tests that depend on it working just so. This is exacerbated by behavioural unit tests, where part of your test also verifies the interaction of a class's collaborators (usually mocked out).

Integration/acceptance testing, on the other hand, is a hugely important part of software quality, and in my experience you should spend a significant amount of time on getting them right.

A lot of shops have drunk the TDD kool-aid, but as the link above shows, a number of studies shows that its benefit is inconclusive.

7
  • 10
    +1 for the point on refactoring. I used to work on a legacy product that had been patched and kluged for over a decade. Just trying to determine the dependencies for a particular method could take the better part of a day, and then trying to figure out how to mock them could take even longer. It was not unusual for a five-line change to require over 200 lines of test code, and take the better part of a week to make.
    – TMN
    Oct 14, 2015 at 17:39
  • 3
    This. In MainMa's answer test 4 should not be done (outside of an academic context), because think about how that would occur in practice... if a person's name is close to the max size for a string something has gone wrong. Do not test, in most cases do not have a code path to detect it. The appropriate response is to let the framework throw the underlying out of memory exception, because that's what the actual problem is.
    – Móż
    Oct 14, 2015 at 20:59
  • 3
    I'm cheering you on until "no need to test those lengthy .equals() methods generated by Eclipse." I wrote a test harness for equals() and compareTo() github.com/GlenKPeterson/TestUtils Almost every implementation I have ever tested was lacking. How do you use collections if equals() and hashCode() don't work together correctly and efficiently? I'm cheering again for the rest of your answer and have up-voted it. I even grant that some auto-generated equals() methods might not need testing, but I've had so many bugs with poor implementations that it makes me nervous. Oct 15, 2015 at 14:27
  • 1
    @GlenPeterson Agree. A colleague of mine wrote EqualsVerifier for this purpose. See github.com/jqno/equalsverifier Oct 18, 2015 at 17:26
  • @Ӎσᶎ no, you still have to test unacceptable input, that's how people find security exploits.
    – gbjbaanb
    Oct 20, 2015 at 13:51
14

Can not be generalised.

If I need to implement a formula or algorithm from physically based rendering, it can very well be that I spend 10 hours on paranoid unit tests, as I know the slightest bug or inprecision can lead to bugs that are almost impossible to diagnose, months later.

If I just want to logically group together some lines of code and give it a name, only used in file scope, I may not test it at all (if you insist on writing tests for every single function, without exception, programmers might fall back to writing as few functions as possible).

1
  • 1
    This is a really valuable point of view. The question needs more context in order to be answered fully.
    – CLF
    Oct 15, 2015 at 21:43
5

I find it to be the most important part.

Unit testing is not always about "seeing if it works right", it's about learning. Once you test something enough it becomes "hardcoded" into your brain and you eventually lower your unit testing time and can find yourself writing entire classes and methods without testing anything until your done.

This is why one of the other answers on this page mention that in a "course" they did 90% testing, because everyone needed to learn the caveats of their objective.

Unit testing is not only a very valuable use of your time as it literally enhances your skills, it is a good way to go over your own code again and find a logical mistake along the way.

4

Is it normal to spend as much, if not more, time writing tests than actual code?

Yes, it is. With some caveats.

First of all, it is "normal" in the sense that most big shops work this way, so even if this way was completely misguided and dumb, still, the fact that most big shops work this way makes it "normal".

By this I don't mean to imply that testing it is wrong. I have worked in environments with no testing, and in environments with obsessive-compulsive testing, and I can still tell you that even the obsessive-compulsive testing was better than no testing.

And I do not do TDD yet, (who knows, I might in the future,) but I do the vast majority of my edit-run-debug cycles by running the tests, not the actual application, so naturally, I work a lot on my tests, so as to avoid as much as possible having to run the actual application.

However, be aware that there are dangers in excessive testing, and specifically in the amount of time being spent on maintaining the tests. (I am primarily writing this answer in order to specifically point this out.)

In the preface of Roy Osherove's The Art of Unit Testing (Manning, 2009) the author admits to having participated in a project which failed to a large part due to the tremendous development burden imposed by badly designed unit tests which had to be maintained throughout the duration of the development effort. So, if you find yourself spending too much time doing nothing but maintaining your tests, this does not necessarily mean that you are on the right path, because it is "normal". Your development effort might have entered an unhealthy mode, where a radical rethinking of your testing methodology might be necessary in order to save the project.

3

Yes, it is normal if you're talking about TDDing. When you have automated tests, you secure the desired behaviour of your code. When you write your tests first, you determine whether existing code already has the behaviour you desire.

This means that:

  • If you write a test that fails, then correcting the code with the simplest thing that works is shorter than writing the test.
  • If you write a test that passes, then you have no extra code to write, effectively spending more time to write test than code.

(This does not account for code refactoring, which aims at spending less time writing subsequent code. It is balanced by test refactoring, which aims at spending less time writing subsequent tests.)

Yes too if you're talking about writing tests after the fact, then you will be spending more time:

  • Determining desired behaviour.
  • Determining ways of testing desired behaviour.
  • Fulfilling code dependencies to be able to write the tests.
  • Correcting code for tests that fail.

Than you'll spend actually writing code.

So yes, it is an expected measure.

2

It can be for many people, but it depends.

If you're writing tests first (TDD), you may be experiencing some overlap in the time spent writing the test is actually helpful to writing the code. Consider:

  • Determining results and inputs (parameters)
  • Naming Conventions
  • Structure - where to put things.
  • Plain old thinking

When writing tests after writing the code, you may discover your code is not easily testable, so writing tests is harder/takes longer.

Most programmers have been writing code a lot longer than tests, so I would expect most of them to not be as fluent. Plus you get to add the time it takes to understand and utilize your testing framework.

I think we need to change our mindset about how long it takes to code and how unit testing is involved. Never look at it in the short-term and never-ever just compare the total time to deliver a particular feature because you have to consider not only are you writing better/less buggy code, but code that is easier to change and still make it better/less buggy.

At some point, we're all only capable of writing code that is so good, so some of the tools and techniques can only offer so much in improving our skills. It's not like I could build a house if I only had a laser guided saw.

1

Is it normal to spend as much, if not more, time writing tests than actual code?

  • Yes for writing unit-tests (Test a module in isolation) if the code is highly coupled (legacy, not enough separation of concerns, missing dependency injection, not tdd developed)
  • Yes for writing integration/acceptance-tests if the logic to be tested is only reachable via gui-code
  • No for writing integration/acceptance-tests as long as the gui-code and the business-logic is separated (The test does not need to interact with the gui)
  • No for writing unit-tests if there is seperaton of concerns, dependency injection, code was testdriven developped (tdd)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.