5

Automated testing is pretty hyped-up in recent years, with particular emphasis on TDD at the "unit" level. The touted advantages include things like:

  • Stabilizing existing code: breaking changes are identified before deployment
  • Coercing code, to some extent, into smaller, more testable structures
  • A codebase that self-documents requirements
  • A codebase that self-documents the usage of its units and modules

The perceived disadvantages include things like:

  • Increased initial development time
  • Increased lines of code per requirement (test code + implementation)
  • The patterns required for certain features to be testable is, in some cases, said to drastically increase complexity / lines of actual implementation code required per feature
  • Increased developer skills requirement
  • Increased tooling

There are also some simple limitations and impediments:

  • Tests cannot ensure code is bug-free; only the specific, test-encoded requirements
  • Tests can only validate testable code
  • Some interfaces and/or hardware may be very/prohibitively time-consuming to mock or fake
  • Some requirements cannot [yet] be tested automatically (e.g., "it looks good")

And so, for a given project, either new or ongoing, how should an unbiased, professional programmer weigh these things against each other to determine whether an automated testing suite is a net gain?

Or, if knowing ahead of time is too fuzzy of a science, having implemented an automated testing suite, can we measure and/or demonstrate its net impact?


For example:

We might conclude "intuitively" that a large, widely used financial system benefits from a thorough automated test suite. Without crunching any number, the cost of adding many meaningful tests "feels" lower than the liability of financial transactions that "forget" which accounts the money is really in.

On the other end, a test suite of any size around a PHP implementation of the echo command is probably going to inflate our development time by 10 or 100 times with no observable gain.

But, what about the things between the extremes?

  • My experience is the automated testing revolution really happened in the early 1990s, at least in commercial software. And now what we have is a rapid development of different ways to use automated testing and particular dogma and excitement around Unit Tests and TDD. So to really answer this question one has to consider what kind of automated testing. – joshp Jan 28 '16 at 23:23
  • 1
    This is one of the places where software development becomes more art than science. Deciding how much time to spend on your tests is a judgment call you have to make based on how buggy and stable the program needs to be, how buggy and stable the requirements are, how inherently testable or untestable certain parts of your application are likely to be, and so on. It's never going to be meaningfully quantifiable. But you can demonstrate its impact simply by changing how much time you spend on tests and then seeing how many more or fewer bugs make it to production. – Ixrec Jan 28 '16 at 23:33
  • @joshp I had no idea it had been around so long! Every day is a school day... – Robbie Dee Jan 29 '16 at 11:23
  • I have to disagree with all the "perceived disadvantages" except maybe "higher developer skill" and even that has much worse consequences than automated testing. – Euphoric Jan 29 '16 at 11:38
  • @RobbieDee Well, I don't perceive them. To me, they come from lack of experience with automated testing. But that might be just my bias. – Euphoric Jan 29 '16 at 11:45
5

For me there is a simple criteria to decide if automatic tests are needed:

Are you planning to evolve and maintain the project? "Yes" - you should write tests.

There are some cases when you can go without tests, mostly it is "do once and forget" type of work, like small web-site, CMS setup, etc. Everything which you do quickly, test if it works and forget about it.

For anything more complex tests are necessary. And there are no disadvantages:

Increased initial development time

This is a very common mistake. Tests decrease development time on any development stage.

If you don't write tests the development process is like this:

  • Write some new code, often touching some old code too
  • Manually test if it works
  • Guess what else can be broken by your changes, re-test it too
  • If you are lucky, you can find something that worked before, but now is broken (regressions)
  • Go fix it
  • Repeat the test / fix process

Besides your main goal (write the new code), you have to do a lot of boring and repetitive work. The amount of this work grows really quickly as project evolves.

What is even worse is that you have to do this boring repetitive work and you still can not guarantee that you didn't break anything that worked before. You can not guarantee that you well tested all the cases for the new code.

Bugs and regressions will go to production. For the user it is really annoying to see the updates when new features don't work as expected and old features are broken. And these bugs will eat even more time - users, support, managers and developers, all will spend time on finding and fixing these bugs.

Compare this with the scenario where you have tests and add tests for any bug you fix:

  • Write some new code, often touching some old code too
  • Add tests for the new code
  • Run the test suite
  • Fix errors
  • Repeat until it passes

The test / fix cycle with automatic tests will be way faster than manual testing and it will actually re-test everything what was working before. You can't even dream of this with manual testing.

The only additional step is to add tests for the new code. It will take much less time than all the additional efforts and problems you have with manual testing. Plus it is not boring nor repetitive - this is something what programmers should do and can do - write the code.

Yes, you still can't guarantee that there are no bugs. But you are sure that all the existing features work and that the new feature works too (at least those cases you added new tests for).

Increased lines of code per requirement (test code + implementation)

Maybe, but lines of code doesn't mean anything when we know that tests save the development time.

For example, you need to implement a popup calendar on your website:

  • Programmer A cares about lines of code and writes the calendar in pure JS in 3000 lines of code and it takes two weeks
  • Programmer B is careless, he gets jquery + jquery ui + some plugin and adds 50000 lines of code, it takes 3 hours

What do you prefer?

The patterns required for certain features to be testable is, in some cases, said to drastically increase complexity / lines of actual implementation code required per feature

Again, this is a mistake.

Same patterns required for testing also improve your code structure. Testing makes you split the code into small, reusable and independently testable units and it is only a win for the general code structure.

Increased developer skills requirement

Not exactly true. Even if you have beginner developers who have to write unit tests, they also will have to learn how to organize their code in better ways and will improve as programmers.

If they don't have to write tests, you'll get a whole project implemented as a solid mess, which will fall apart really quickly. Adding a new feature will generate 10 new bugs and at some point it will be not possible to add new features at all.

Increased tooling

I am not sure I understand this point.

Do you mean that you will have to use some Continuous Integration server? This is a good thing. It will help you shorten the "develop-release" cycle and you'll have no fear giving you users the new version.

Do you mean some infrastructure to write and run tests? There are plenty options available for every programming language.

Regarding the general limitations of the unit tests you mentioned: these are real and you'll want to solve these once you start writing tests.

But that's not the reason to go without tests at all. Write tests where it is possible, manually test the rest and you'll quickly find some ways to add tests for the rest too, just to avoid this boring chore.

  • "Tests decrease development time on any development stage." Sorry, but that's a ludicrous claim. If a test suite is part of the project, part of the project setup necessarily includes finding, installing, configuring, or otherwise preparing a tool (to answer your tooling question) to run the tests. And then you need to actually write and validate of the tests. Is it faster to write a PHP implementation of echo and just run it to see if it works? Or to break the implementation into units, write tests around them, red-green the tests, and then run it? – svidgen Jan 29 '16 at 15:00
  • In general, "higher complexity" may be a criterion, as well as plans for the application's future. But, your other points are largely just blanket dismissals of perfectly rational concerns. – svidgen Jan 29 '16 at 15:02
  • 1
    @svidgen There are some simple cases when you "do and forget", where you can quickly craft it, verify and use. My points applies to the projects which you are going to evolve. Here tests will save time for everyone - developers, managers, support, users. I am really sorry that you find my points ludicrous. This means I failed to translate my thoughts and feelings into the text. For me, every disadvantage from the list, can be counted as a disadvantage only in a really short-term. Once you start looking a bit further, you'll see the benefits. – Boris Serebrov Jan 29 '16 at 15:29
  • Ha! Only the one point was ludicrous! (About initial development time.) Beyond that, I suppose the contextualization makes the arguments less blanket-like. But, defending automated testing isn't really what I'm looking for -- I understand the advantages. What I don't have is an objective way to determine or communicate where the lines between "short term, simple" and "long term, complex" are. – svidgen Jan 29 '16 at 15:50
  • I think your first few statements answer the question. And although I disagree with the statement that "For anything more complex tests are necessary," it's at least along the lines of answering the question -- rather than restating the advantages. – svidgen Jan 29 '16 at 15:52
0

"All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident." - Schopenhauer

At a macro level the argument was had a long time ago with formal testing. The argument was that programmers are highly trained professionals who should write code without defects. This is of course ridiculous as (even if this were true) many external factors come into play:

  • Woolly/changing requirements
  • Changing environment
  • Limited time to test
  • Changes in external systems
  • Different expectations for various users
  • Feature fatigue

The benefit which never seems to get talked about with automated unit testing during development is that a failed test fails early. It is all too easy just fix the test and carry on blithely on your way. But consider how expensive the change would have been to attend to had the software been shipped.

So...your question. Simple answer - you don't. The very nature of software is that no two pieces are the same so you can't always compare like with like. That is something the average layperson doesn't get. They cry: "Why are there bugs?" But the truth is that every new piece of software is essentially a prototype - warts and all.

Something you can measure however is failures in CI server builds. If tests go over then either the developer hasn't run the tests or there was an issue with the merge/build set up. Failures flagged here indicate failures that would have happened further down the road when they would have been potentially more expensive to fix.

  • I don't think this really answers the question -- even if the answer is intended to be, "the question can't be answered." And in regards to the last point, yes, you can measure (count) build failures (or regressions, or bugs overall, or anything that you can count), but the count is meaningless without a comparison or control. – svidgen Jan 29 '16 at 15:55
  • I can't vouch for all CI servers but in TFS at least, you can keep history for as many builds as you like (or have space for). – Robbie Dee Jan 29 '16 at 15:57
  • Well and good. But, what metrics with respect to what are you looking for in your CI server stats to communicate the net effect of your testing practices? – svidgen Jan 29 '16 at 16:18
  • That would rather depend on the severity of the bug that could occur on the back of the failed test(s). – Robbie Dee Jan 29 '16 at 16:19
  • So, you'd need to assess every single potential bug and assign a severity measurement?? How do actual bugs factor in? Or feature-not-implemented and features-not-even-imagined-yet due to the time spent fixing CI build failure that occurred due to test failures on non-critical features or business rules? – svidgen Jan 29 '16 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.