How important is automated testing in rapid release, non-critical (web) apps?

Question

I think I understand the theoretical benefits of automated testing, especially unit testing. However, I'm not sure what the optimal amount of testing is when the project is a non-critical, rapidly developed (and deployed) web app with a reasonably large user base. My current reasoning goes like this:

• Most of the core functionality of our app can be manually tested in a matter of seconds. (The site is a type of search engine, so basically - run search, get results, it works.)
• Of course, there are many edge cases and options and such that could be tested, as well as unit testing of underlying logic, but if anything significant breaks and isn't caught before it's made live, we literally tend to get email from users within an hour. (Within minutes if it's a major problem, but very rarely does one of those get out.)
• So the consequence of any significant problem going live is that a few people are inconvenienced for a few minutes. Maybe some percentage of new visitors we get in that period would be lost, although the numbers should be very low since the time period is so short.
• Lower impact bugs might take longer for someone to email about - days, or even weeks depending on the severity - but they will also have less of an impact on users, so from a business perspective likely don't "matter" as much.

Now, I'm not saying that excellent test coverage is useless. Of course it would be nice to catch every issue before it goes live. But in the cost/benefit tradeoff between working on tests and responding quickly to user feedback / adding and refining features, the latter seems to win.

The one exception I've found is in code that doesn't directly affect the user. For example, ad serving code, or data collection. These don't directly impact the user's value from the site, so people are unlikely to let us know when they break. They will also be less noticeable in our feature testing. So I can see a strong case for well-defined unit tests there. (And from there, there ironically seems to be a spectrum where good test coverage becomes less important as features become more important, to the point where code that is critical to the site's operation doesn't really need to be tested (in a well defined manner) at all.)

So my question is, am I way off base here? Is widespread test coverage and/or test-driven development optimal even in a non-critical, continuous-release environment? And if so, where am I failing in my reasoning?

Or, assuming we fix bugs immediately when reported, is it reasonable to focus on rapid, nimble development, knowing that our users will let us know if we break something? (With the exception that we will have formal tests for functionality that is not directly exposed to the user.)

Answer 1

To be honest, these days I am more and more inclined to simply comment on this as unprofessional behavior. There are very good books like Legacy Code 1, Refactoring [2], or Clean Code [3], which explain much better why professional software development should always include tests.

Warning: the following are my opinions and sometimes phrased rather strongly, because I do strongly believe them, not because I want to put you or your project into a bad light.

Advantages

You gave a few reasons, why one might want to add tests, however, you seemed to have missed a few details on those.

Code that doesn't directly affect the user

This is a double-edged sword at best, or just plain pointless. If it doesn't affect your user, then it's simply by the very definition of the word useless. Every piece of code should affect your user, because it is the reason why your user wants your app at all. Of course, you meant it in the sense of code that relates to the visually observable aspects of your app (aka GUI parts). But that means we're discounting such a small portion of your overall codebase, that it's not really worth discussing. It's like having a warehouse full of plums and wondering whether it's worth it to buy a corer for the plums at all, because it may not even work for that one apple in your hand.

Consequence is inconvenience

You are quite correct in that existing users are inconvenienced. But I do hope you are the market leader, or even better the monopolist. If there is a second app available to the user which does not inconvenience her - guess what? she'll switch. Often this happens faster than you can say "I'll fix it". Obviously, newcomers to your app will just throw it into the trash right away.. it's not working at all when they try, so why bother? However, apart from simply losing direct sales on your app, there is another thing at stake: your company's reputation. What do you want people to think and tell others about your company? "These guys know what they're doing. Almost never do I have an issue with their products and if I do, they fix it asap", or do you prefer "I have no idea what these guys are doing.. every once in a while their products just stop working for a few hours. They should have fixed that long ago, but somehow they just can't get it right". The more impact your products have, the worse the problem. As a rather extreme case look at something like the Diablo 3 game, which was not functional for the first week after its release due to the massive demand of players. What did function very well though was the users venting their frustration everywhere on the net.. even today, the game has not exceeded 3 stars on amazon - granted, there are other issues with it, but the name Blizzard is marked in players' heads as "do not even expect to play on release dates. they'll screw it up again.".

Complexity of test suite

(you mentioned that one in your comment) Oh so true, more complex software means more complex tests.. or maybe not. This is a typical fallacy. Stop believing this. It is on the same line as "Sht in, sht out".. if your software is ever-so-complex, then tests or their complexity are not your problem. It's simply that software's complexity. Increased test complexity is just a symptom of that original problem. Tackle the real problem and make your software less complex, and lo and behold, your tests don't need to be complex either. It is also a very short-sighted argument that you lose time due to having to update tests when you make changes. Look at what really happens: you change the code and a test fails. Duh.. stupid tests.. but really, this just told you that you changed the code in a way that also changes the previously specified behavior of your program. Were you aware of that even? If not, then stop complaining right now, because you have been acting carelessly! If you do want to change the specified behavior, then tests even ensure that you did just that. What if you want to change from previous behavior to a new one, so you change the code, but no test fails? Oh oh.. major problem spotted right there. Complexity is a beast and truly unavoidable overall. But tests are the cage that keep it at bay and help you deal with it in a controlled manner.

And several more you missed:

Legacy code

Most developers consider the code you are writing for your app today as legacy code. In 1 legacy code is even defined as code without tests. So what are the problems of legacy code? oh right.. that one guy which knew the code was just hit by the bus (aka left the company for a better opportunity). I bet your successor would contemplate giving his left foot for solid test coverage when he realizes that he now has to maintain someone else's software which goes into production to thousands of users, which even expect bugs to be fixed by the hour.

Refactoring

You can't do that. Period. The evolvability of your software is crippled to the point of non-existence. Everything you add to the product, every bug you fix, every minute you invest into that code, translates to even more costs further down the road. There is no risk-free clean up and there quickly comes a limit when you will no longer be able to just say "oh let's rewrite that.. if we mess up we'll hear from our users soon". Since you have no evolvability and refactoring, it just snowballs from there.. code complexity will rise together with bug counts and costs, each trying to out-do the other. Of course, this is no issue for you and your software, because it's small, simple, easy to maintain - step back just one step and look into the mirror. When you tell that to others what they hear is "famous last words". Stop fooling yourself if you really still believe that. In my experience, the only software that does not grow is the one that has utterly failed all of its assigned tasks - and you have bigger worries than testing in those projects.

Development speed

Every fix, every new feature, every single line of code you add or change, profit from tests. Combined with active refactoring of your codebase the development speed surpasses a project without tests very very quickly. The keyword here is feedback loop. While the feedback loop you have described via your customers is indeed a very short one, which is good!, it is also a very costly one (see above). Tests simply mean that you get an earlier feedback loop that is shorter and cheaper.

• Multi-Project view: When working on source code in a company, your project seldomly works in isolation. Projects can cross-fertilize each other by re-using or sharing common code. While this does add complexity and gives your architects headaches, it is all based on the very foundation of testing. Do not even dare to think about re-using or sharing code that is untested - you'll sacrifice both projects on the altar of ignorance that way.

Regression

Even though you can fix bugs within an hour (I do certainly not admire your sleeping pattern if that is for real!), your customers MAY (yes, that is a huge may) accept that for new features and previously unknown bugs. It is simply unacceptable and projects an immensely bad picture of your company, if the same bugs appear again lateron. If you start fixing a bug without writing a test to verify its existence first, then you are making a huge and costly mistake. If you cannot write a test against the bug, then you definitely are in no position whatsoever to even attempt to fix it. And if you wrote that test, you immediately see when it is fixed (because the test should of course assert the correct behavior, hence, turn from red to green due to the fix). Finally, the very same test serves as a regression test from now on to ensure your users will never ever see that identical bug again. (Off-topic: I'm sometimes wondering if it's worthwhile to introduce a saying like "Write once, profit everywhere" to capture this..)

Professionalism

What do you tell your manager? Do you have any moral conflict in confidently stating that you made this piece of software? or do you feel slightly ashamed, knowing that you just hacked it together and most of your confidence of the software's capabilities relies on sheer hope? Have you really done the company a service, or did you just create further obligations it has to pay for dearly in the future?

The other side of the coin

While I do argument in favor of automated testing, it was correctly pointed out, that this answer was one-sided, so let's take a look at the cons of automated testing.

I have written another lengthy answer that goes into more detail on the general disadvantages of all automated testing. But in short, these disadvantages turn into advantages when you do not implement automated testing. Here's that same list in its "advantage"-point of view formulation:

• Development time: You get faster time-to-market
• Skill level: Your developers don't need to know anything about automated testing.
• Tooling: You don't need these tools or licenses.
• Failed and non-failed tests: You are not wasting time on tests anyways.
• Deployment costs: Costs are reduced to what you need for operation. Testing infrastructure cost is no longer needed.

Summary

In the end, its up to a lot of factors on what the best approach is. I would argue that there is no objective answer that holds for all projects and is concrete enough to implement.

You can of course say that there are costs/profits (positive and/or negative) over time and you can think of two mathematical functions, one for automated tests, one for development without them. At the very least, there will be an intersection somewhere after which if you spend more time on the project, automated tests will have a better overall cost/profit. That point is definitely too late for a switching strategy though.

Personally, in my experience, people severely over-estimate how long it takes to reach the point when automation pays off - myself included. Even projects that are supposed to only take 2 weeks have that tendency to keep going. Since the automated tests already help during development, the time delay caused by adding automated testing is similarly over-estimated. I am still unsure at what point automated tests start to pay off, but in my past that point was reached already a week or two into the project. In some projects the tests were there and a retrospective revealed the benefit, in other projects problems soon (and later) started to come up that would have been avoidable.

I'd suggest to dismiss the notion of automated testing yes-or-no, and instead have automated testing and think of it as a tradeoff trying to find out just how much of it you need.

1 Michael Feathers' Working Effectively with Legacy Code (ISBN 0-13-117705-2)

[2] 1999. Refactoring: Improving the Design of Existing Code, With Kent Beck, John Brant, William Opdyke, and Don Roberts (June 1999). . Addison-Wesley. ISBN 0-201-48567-2.

[3] Clean Code: A Handbook of Agile Software Craftsmanship. Prentice Hall PTR. 2008. ISBN 0-13-235088-2

Answer 2

This depends on how good you are at writing tests, as well as, how much you want to maintain and improve your test writing skills. If you're not good at it and don't care to learn, it could increase the amount of time wasted.

Of course you may think having to write tests will slow down your rapid release cycle, but at some point the application may become so complex that having the tests will save time in the long run.

You can create all types of scenarios where it appears you can avoid following sound coding habits, but they rarely exist. How much coding can you get away without having source control? It's like not requiring a strong foundation for a house because you don't plan on having a lot of people in it at one time. Cutting corners will probably cause you to end up spending a disproportionate amount of time fixing a bug. Wouldn't that really be the true waste of time on a non-critical app?

Answer 3

To answer directly, unit testing is critical, even for non-critical web applications. Have you considered what vulnerabilities exist in your search engine? There may be issues in your web app that some customers have noticed and not provided notification. Unit testing at the very least can check for input validation (or lack thereof) in all layers of your software.

Another thing to consider is how much you value your own time as a developer. Fixing bugs that could have been caught with simple unit tests, or making re-fixes because one defect affected another are known time-wasters. Take the effort to write unit tests.