5

This question already has an answer here:

I am the sole maintainer of a code base of about 2000 lines of code. It's not big, but over time the code became so unmaintainable my boss agreed to give me time to rewrite it from scratch.

Since the code became such a nightmare to maintain and extend, I wanted to do things right this time and thought about writing unit tests from day 1 (the current code has no tests). However I'm uncertain about this.

On one hand, I estimate the code base will probably never exceed 2000 or 1500 lines of code in size. On the other hand, this app is estimated to be in use for a while, I assume at least for 6 months.

So the question is: What size of code base justifies writing unit tests? Is unit testing overkill for a project of this size? Or do the reasons to write unit tests for larger projects apply here as well?

Please note that I am aware of the great advantages of having unit test coverage of the code base. I'm simply wondering whether this would be an overkill for such a small app.

marked as duplicate by yannis Dec 4 '15 at 7:49

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 5
    It's not the size of the code-base, but the complexity of the various flows through the code. – FrustratedWithFormsDesigner Dec 4 '15 at 3:11
  • No code base is too small for unit testing. I have unit testing for each library function, some are only a few lines of code. – Rohit Gupta Dec 4 '15 at 20:33
10

Is it overkill to write tests for a small code base?

Absolutely not.

Code bases have a way of growing on us. You've seen this play out first hand already. What was a few hundred LoC grew into a few thousand. A few thousand can easily grow into tens of thousands.

Have you ever heard of the broken window theory? If you start with tests, it's more likely that more tests will be written as functionality is expanded and bugs are fixed. If they're not there, people are unlikely to write them later (not to mention that they'll be harder to write later and won't cover as many cases).

So, yes, write the tests, but I'd urge you to reconsider completely rewriting the code from scratch. You'd do much better to pick yourself up a copy of Michael Feather's Working Effectively with Legacy Code and putting it under tests before and while you're making changes to it. It's a small project, but that doesn't mean that it can't/won't suffer the same fate as Netscape.

17

We do unit tests to ensure (*) the correctness of code.

(*) to ensure: in a colloquial, not mathematical sense.

The question of correctness of code applies regardless of whether you have 5 lines of code, 2000 lines of code, or 200000 lines of code.

I mean, a binary search routine is 10 lines. You would be insane to not unit-test it.

So, the choice is yours. (So to speak.)

If we are to take "small codebase" to mean "correctness is unimportant", then sure, don't test anything. But there is nothing that equates the two.

If correctness is of importance, then it is advisable to test at least those portions that are crucial for the correctness of the application and that are hard to test by simply running the application. And there will always be those. (More discussion about that in the comments below.)

  • 1
    @AvivCohn why would hand-testing be enough? Can you test by hand whether a binary search routine embedded somewhere within your 2000-line program works correctly for all fringe cases? Can you test by hand whether a particular exception is thrown under specific error circumstances? And, fine, you could perhaps be inhumanly meticulous and do it once. The second time around you will be bored senseless to be that meticulous again. – Mike Nakis Dec 4 '15 at 1:27
  • 1
    Actually, Aviv, if correctness is of sufficient importance, I would not even agree that it would be an overkill to write a test for a "guess the number" game. For example, one thing that we could test for is uniform distribution of the numbers to guess within the stated range (1 to 10) – Mike Nakis Dec 4 '15 at 1:35
  • 2
    @AvivCohn I believe the question is the 2-dimensional size of the app: (LOCs, Years to stay active). I guess a throw-away script can do without unit testing :) – Elazar Dec 4 '15 at 1:36
  • 1
    That's what I intended to mean by "the choice is yours". If you need correctness you have to test even the 10-line function, because even that can have subtleties that a human might not be able to detect by simply trying various values. Perhaps I ought to have been more clear about it. – Mike Nakis Dec 4 '15 at 1:36
  • 1
    And of course in the "guess the number" game I would only test the tricky thing, which is the uniform distribution, I would not test whether the game is capable of playing rounds correctly, because yes, that would be immediately obvious, not only by only a couple of rounds of playing, but also by just looking at the 6 lines of code. – Mike Nakis Dec 4 '15 at 1:45
3

Nope, test the little ones, test the big ones - even test the ones you copy directly from the text book.

Joshua Bloch, who wrote the sort for java.util.Arrays, tracked down a bug which sat in one of the Java standard libraries for 9 years, and in a text book implementation for about 20 years.

It's very subtle, so likely affects a large portion of implementations of divide and conquer algorithms (at last as of 2006, it's a nice fantasy that this is probably no longer true).

Test everything you can.

2

Before you think about unit testing, you need to look into why the code is a "nightmare to maintain and extend".

It is very possible that there are a lot of undocumented changes in it, therefore causing difficulty. You MUST absolutely be 100% certain that your rewrite will not ignore currently implemented, but undocumented, patches to this code. Your unit test will be 100% useless until you figure out exactly what your code need to do.

Also, try this article regarding software rewrites.

Now specifically about the question "What size of a codebase 'justifies' writing unit tests?" There is never overkill for unit tests, as other answers have tested.

Even regular expressions, usually only a single line, requires unit tests. My unit tests for my regular expressions are at minimum 10 distinct inputs, and then some more time is spent to specifically break them.

  • 1
    Good advice, but not an answer. – Elazar Dec 4 '15 at 2:15
  • And funny, the artice you linked is basically a rant about how hard it is to rewrite a non-tdd-style code. "Those are bug fixes" that no one is aware about and there are no regression tests for them, so they are likely to reappear, or are now obsolete like the client configuration they target. – Elazar Dec 4 '15 at 2:23
  • 1
    The only detail on the rewrite is that it is "unmaintainable", either the OP forgot what he did and is redoing it, or he is maintaining someone else's code. Either way, the article covers why rewriting is not a solution to the correct problem, because the problems were never identified. – Nelson Dec 4 '15 at 2:25
2

Every significant decision in engineering, or business in general, or life in general, can stand to gain from a COST-BENEFIT ANALYSIS. The secret to success is achieving maximal efficiency; we all have limited resources and don't want to compromise quality. "Develop tests for only that which will benefit from them." As professionals, we get paid to make value-creating judgment calls like that.

There are no simple formulas. A 10-line function that does cryptic math calculations will probably benefit from a unit test. A 1,000-line set of formatted output statements probably won't.

The same holds true for the number of tests per unit. Some units may benefit from 40 different test cases. Others may benefit from no more than 4.

Think of taking a standardized exam. Does it make sense to check all your answers twice, 3 times, more? If you finish the exam in an hour but are allotted 4 hours, is it worth it to you to check each of your answers 8 times? How many errors would you expect to catch in an 8th round of checking? At some point, you call it a day and turn in your exam.

-2

A bit lengthy of a post here, but the whole testing thing has been giving me an itch lately...

It really depends on the nature of the code. Testing as religion is absolutely stupid. Never testing code that is 90% pure, side-effect free, well-bounded functions is equally stupid. Never writing tests as an exercise to help you understand what the bounds of a pure function should be can leave your code in a state of limbo where you don't really understand what it does, how it should be structured, or where the corners are.

The third case is the real reason I write tests on old code. We already know it works well enough, but until I mess around with it I will never fully grok it, yet I am the maintainer. See the problem? Testing is an opportunity in this case. It forces me to understand it, and understanding it allows me to see where I can factor out side effects, create pure functions that truly are testable, and transform the code into something inherently testable, well-bounded, and full of explicit semantic meaning the next maintainer will grok immediately.

Why would you not do that? (A: Because the world is on fire, the company is in the red and you have bigger fish to fry just then... (note that this can also be a reason for doing tests))

But I'm not talking about writing tests for testing's sake -- that is a fad gripping the industry right now, replete with buzzwords, consultancies, methodologies based entirely around it, etc. And yet software still has, well, just as many bugs as it used to.

To quote from Rich Hickey:

What does every bug in the wild have in common? They passed all the tests.

That joke is on us.

But these words: "pure", "testable", "bounded"... what do they mean?

When I talk about pure, testable, bounded functions, I mean functions that do not depend on any external resources or shared mutable state and provide a return value that we can test. No file descriptors are being passed around, no socket of indeterminate state is coming in or out, we're not trying to get control of a camera or GPS device that may or may not exist, and we're (hopefully!) not dealing with a variable a or b that is referenced by other concurrent threads without being locked which might change randomly in the middle of the operation.

With a function that has an input and an output, we can write a test and actually know something about it. Writing a test for an entire object, on the other hand, becomes somewhat more problematic -- especially in a concurrent system. Several threads may be referencing that object at the same time, internal state that the return value of a method call depends upon may change unexpectedly halfway through execution unless the relevant memory locations are locked -- and if they are not released other problems occur.

Managing the complexity of the interactions among threads can be a real mess and renders quite a bit of OOP unit test code useless because method calls that update internal state were never made reentrant, and often places where a serialized communication scheme among threads is necessary are overlooked until the system has just crashed enough in production to "shake out the bugs".

In this environment an obsession with testing can turn a frustrating death march into an endless waking nightmare -- especially because in many cases the test environment influences the nature of the bugs present. That's when you get the fatal mix of "my testing environment suppresses bugs, so I can't find them", or, the even worse "the production environment works fine, but half the tests fail... wtf?" This problem is especially prevalent in short programs, because short programs usually exist for the express purpose of interacting with an external resource.

We forget that accessing external resources is another form of concurrency. Even in single-threaded code external system calls to do things like refresh the screen, get input device status, open a file, flush a network buffer, etc. are executed on their own schedule, sometimes independent of the executing thread (not all system calls block -- unless they do).

Instead of writing functions that open a file, check for some value in it, pass the file descriptor on to another function that appends it, then passes the file along to another function that does something else, collect the operations you want to do with that external resource in a single function, and call pure ones to get your values. (Protip: This is the magic that underlies the monadic movement in "pure fp". Its not as much about mathematical monads as formalizing a way of making batches of prayers to the system gods.)

It isn't always possible or practical to aggregate side effects for execution in a limited number of places, but refactoring with a clear goal of aggregating side effects can transform most of your codebase from a mysterious mess where you have 100% test coverage, but still >80% of the code's state is indeterminate at any given time. Flipping that around to where 10~20% of your code is where the side-effects happen, and the rest of the code doesn't touch the outside world leaves you in a much better situation.

Even on a tiny codebase this is can be a big win. ~2000 lines? Why wouldn't you do this? After all, silly bug was found in the canonical implementation of binary search after years of use.

On the other hand, a commitment to testing is, above all, a business decision. If you have a lot of stuff on your plate already, and those things are priority targets your organization needs to hit to satisfy some high priority goal and your tiny 2000 line program is working fine already -- then the testing can wait. The bug in binary search existed for about 30 years (and still does in most old code that is happily running in millions of systems across the world...) and yet nobody's hair has caught fire because of it.

As with so many other things in life, with testing you must seek balance.

  • 1
    Commitment to testing, IMO, is absolutely not a business decision, much like deciding on your application architecture is not a business decision. Especially since, in this case, it's greenfield project. – ZenMaster Dec 4 '15 at 3:53
  • 2
    @ZenMaster Everything is a business decision. The business is what pays for the party. If your business decision makers are so distant from your technical decision makers, and yet your core business is software then you have other problems. Even in internal code for a non-technical company any coding decision is a business decision within the scope of the resources available for the person or department doing the coding. The only time this is not true is when you are writing code for fun or volunteering on an open source project -- time/resource constraints are arguably tighter there. – zxq9 Dec 4 '15 at 3:59
  • 1
    You don't fit development practices to the budget, but you absolutely fit dev practices to the risks of the software. Do you write 2,500 pages of spec for a 6,300 line change? If not, you've compromised on quality because that level of rigor doesn't make sense for your business. – Alan Shutko Dec 4 '15 at 5:05
  • 1
    @ZenMaster Since when is documentation not part of "development practice"? Since when have all integration tests been verifiable? There are limits. Businesses flex to meet both budget and deadlines. That said, I'm pretty sure that you and I would simply choose to not stick with a place that was getting really out of hand. But every time you write a unit test that doesn't cover every possible state of a unit (which is almost always if it has side effects or non-trivial internal state), you are compromising on quality but it is by necessity. The perfect is indeed the enemy of the good. – zxq9 Dec 4 '15 at 6:41
  • 1
    @ZenMaster I should mention, by the way, that this is a big part of why I now generally write only single-threaded, tight, small programs or massively concurrent, strict message passing, crash-fast large programs. That middle ground is just awful, especially for trying to figure out if tests and specs are verifiable. – zxq9 Dec 4 '15 at 6:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.