Why are my unit tests so expensive? [duplicate]

Question

I am a Java developer, working on a small project. We are three people in the team, we have a budget for 3 months of coding (+ some time for the Analyst, Project Manager and QA team). It is a small project, the budget is really limited. There is some support in our organization to do the unit tests, but I didn't meet any expert in this area. Motivated by books and discussions, I started to do the unit tests. I have some limited experience with frameworks and approaches, so I don't test every getter, I focus on business code, I use Mockito and MoreUnit Eclipse plugin to generate the skeleton and Mocks. I design my code upfront, I use plenty of design patterns, I keep my code clean, with short methods, I respect the Single Responsibility Principe and I have much more public methods then the private ones. It is not perfect, but it is the best I can do after 5 years of commercial experience in one corporation and one mid size company.

So why does making unit tests take me the same or even higher amount of time then coding productive code? After last two days of unit testing I found just one minor bug and it didn't come from the tests, I just randomly found it when I was looking into the production code. Where are the benefits the people are talking about? Every refactoring caused just throwing away existing unit tests and they never told me if I influenced the rest of the system. Every class I test has mocked out its surroundings, so if I change one class, it will not affect the other tests and I won't know if I messed up the system by my change.

What am I missing? Am I doing it wrong?

Edit: To add some numbers, I spend about 80% of time by designing and coding, and 20% by unit testing, so just small amount of my code is covered by tests. I try to do unit test fot 1,5 month, usually after I finish some functionality and roughly test in on my local development machine.

Answer 1

If you first test things manually through a user interface, until you do not find any bugs any more, and write unit tests afterwards, it is not very surprising that those unit tests do not reveal many bugs. You are probably just automating things you already have done manually, but do not code many test cases which were not already part of the manual tests.

Unit tests and other automated tests will show up their value much clearer when you

• can use them for testing things which you cannot easily test through the UI
• when you develop small, UI independent components and test them through unit test without actually having coded the UI for them
• when your application becomes bigger, and after release no. 25 or later you notice that repeating the same tests again and again through the UI becomes tedious
• when you develop software without any UI or where the UI lets you only access a small part of the system directly (for example, a google-like UI).
• when you do TDD, where you must write unit tests, otherwise you cannot even start with writing code

So if you are not going to switch to TDD style coding, and as long as you are writing UI applications up-front, with most functionality directly accessable through the UI, do not be astonished that test automation of any kind will be an additional investment, which may look tedious to you and will not show its real value immediately.

Answer 2

One thing to remember about testing is that you're going to do it either way. Whether you're printing application variables to the screen, using a debugger to inspect your application's state or just visually verifying the final output of the application's input, at the end of the day, you'll have spent quite a bit of time commenting out code, perhaps setting breakpoints or performing other testing tasks in an attempt to verify that your code is functioning within the expected business rules. The question to ask yourself is, will you codify the fruits of your labor in a repeatable process, or simply delete/comment out the ephemeral test code once everything appears to function correctly?

The benefits of testing become more visible the longer the project goes on. When one first designs a piece of software it might feel like tests are unnecessary because it's easy to imagine many of the edge cases in one's mind, however, as a project matures and becomes more complex, you'll eventually encounter unforeseen problems that when solved may inadvertently break something you've already fixed and tested manually. If it's been a long time or someone else ends up fixing the bug, you won't even realize that time has been wasted retracing the path of the problem, assuming the bug is even rediscovered at all. Unit tests let you move forward in confidence knowing you're building on top of a foundation that performs as specified. Creating tests means that every bug you fix is also solved forever, never to be inadvertently reintroduced into a live product through subsequent code changes by yourself or someone else on the team.

Writing tests sometimes feels like more work because you can look at the test and identify it as the product of your time spent working during a session, however, every experienced developer knows that sometimes one can spend hours trying to debug what seems like a very simple problem, only without tests, we simply attribute the time spent to "necessary debugging" rather than "writing tests all day". You can't blame a bug that no longer exists, but it feels natural to look at a paltry set of tests and wonder why you don't have more to show for your time when in reality both outcomes are the same.

Now, this isn't to say that it's not possible to waste time fiddling with test tooling or writing inefficient or unnecessary tests. It does take a bit of time to get a feel for writing appropriate tests but in my experience the best way to avoid these problems is to plan ahead. Define a specification that serves as a definition for a properly functioning baseline or prototype of your application and test to the spec. If you find that the spec is constantly changing and you're always throwing out tests then it probably means that your design is flawed or that the problem space is not properly understood. The key to writing successful tests is having confidence that the tests truly represent the product you want to produce.

Answer 3

Why are my unit tests so expensive?

For the same reason divorce is so expensive:

It is worth it.

In my experience (bizdev, C#/Java over more than a decade now), writing unit tests for code takes about half the time as writing the code itself - so 33% of your total coding time. Some things will be more. Some things will be less. When you're just learning to write unit tests, it will be slower than "normal" coding. That (in my experience) provides the sweet spot for most code between time spent and effective testing. That will usually be a good 70-80% code coverage, give or take. Some environments might demand more rigorous quality. Fewer might need less.

That might seem like a lot, but having the tests saves you time you would otherwise spend tracking down QA bugs, making patches, manually testing, or suffering from code you can't effectively refactor.

so if I change one class, it will not affect the other tests and I won't know if I messed up the system by my change.

Sure you will, if you mocked out the interactions between your classes properly. If you changed your class and your unit tests pass (all inputs are handled as expected, and provide legal/expected outputs) then you can't impact the rest of the system, since the interface between your class and the outside world is intact and well tested.

Answer 4

The key is to test the intention of your code.

This way even if you refactor the way your intention is being implemented you will still be able to have valid unit test because it is still the same intention.

As you have mentioned yourself, if you have limited resources, then do what is most important and that would be business logic.

Also don't forget about acceptance and integration tests since these could eliminate the need for you to be testing each and every layer individually.

If you are following all of those best practices, as you have mentioned above, then writing unit tests shouldn't be that difficult - maybe you are just new to the testing paradigm and with practice you will accelerate and be able to produce code more quickly with better quality.

Answer 5

It may take you longer to write code complete with unit tests. In fact it probably will. What makes it a more productive way of working for the organisation as a whole is all the things that don't happen further down the line, and in particular don't happen to other people, when you have your code armed with a full battery of tests. Fewer manual testers required to achieve the same level of quality. Fewer expensive emergency support calls. Fewer customers who quietly slink off to a competitor without alerting you to bugs. Less time spent bringing a new colleague up to speed, because your test code documents what the production code should be doing. Less time spent mending damage caused by a substandard new colleague.

This view of productivity is hard to measure, though, which is why management and programmers often instinctively revert to measuring the time taken to go from blank screen to something that mostly works, most of the time.

Answer 6

The type of unit testing you need to do to find bugs in existing mostly-working code is very different from the type used as a scaffold while creating or modifying the code. Only the latter is TDD/BDD. And, as it is somewhat newer, it tends to be the one that gets discussed in articles and blogs. Sometimes to the point where people get the impression that it's 'what you need to do', as opposed to 'what you need to do in order to...'.

The underlying justification for TDD, as opposed to other design approaches, is that some of the design scaffolding sticks around and becomes useful tests. Whether or not you buy into that, erecting scaffolding after you have finished development is pretty much never going to be useful.

Some tools, like JUnit and mockito, are common to both approaches (although over-use of mocks will quickly lead to tests that are worthless as anything other than scaffolding). Others tools, like quickcheck and tcases are worthless as scaffolding, and excellent at bugfinding.

Whether or not you use unit testing during TDD, any unit testing you do after the code is complete should be oriented towards bug-finding. Depending on the reliabaility/cost tradeoffs involved, it is sometimes perfectly rational to selectively replace scaffolding tests created during development with bugfinding tests.