Automated Testing: Explaining its Business Value

Question

To start I don't think this is a repeat of other questions on unit testing. What I'm looking for help with is articulating its value to a team of programmers, analysts, managers and testers. By automated tests, I don't think I need to make a distinction between unit tests (e.g. JUnit), BDD (e.g. JBehave, Fitness) and UI (Selenium , Watir) because I think they all provide similar value (but feel free to write an answer that disagrees :))

The following is a list I've identified, I'm looking for answers that help expand or refine:

1. Time/Cost Savings: writing automated tests can take more time than written test cases. However, considering tests are run multiple times, the marginal work (i.e. cost/time) to execute automated tests is several orders of magnitude less. That the automated tests are cheap to run facilitates changing the system over time.
2. Documentation: there is no truer way to know how a system works than its tests. Any other documentation is usually out of date the moment its written, but tests (at least those that pass) reveal how things actually work. This is true for both end-user AND API documentation.
3. Code Quality: test writing forces you to:
   • consider clients because tests are a client
   • breaks dependencies where making code testable often means figuring out how to make that code not require some other large system be available

Answer 1

A few of my thoughts:

1. Be honest that writing automated tests will take more time. If you're doing unit level TDD (which I would recommend as a starting point if you're going to invest in automated testing), you can expect about 30% extra time needed to code a feature. The key here is explaining that this extra 30% (which is probably higher than 30% in the beginning as your team learns how to write good tests) is an investment built to save costs over time. As least with unit level TDD, the design of your system is loosely coupled and highly cohesive, which makes your system adaptable to change over time. New requirements and unexpected bugs always require changes to your system, so keeping your system in a state where change is easy (from the nice design that comes from TDD) and safe (the set of automated tests that continually verify your system) means that it will cost less money to make those changes.
2. There's lots of debate about the value of Acceptance level and UI level tests given the amount of time it takes to write these tests, how long it takes to run them, and how much maintenance they require. I'd recommend reading this article by James Shore about this.
3. In the world of automated testing, there are good ways and bad ways to do it. If you are pitching automated testing to your management, I would pitch alongside it how you're planning on getting your team trained in writing good tests. The Art of Unit Testing by Roy Osherove, Working Effectively With Legacy Code by Michael Feathers, and The Art of Agile Development by James Shore are all great books that deal with these topics directly or indirectly. You should also look into some sort of coach or formal training as well. It's a big change.
4. In terms of Business Value, #2 and #3 of your points above actually serve your first point, so I'd hammer home on point #1 and talk about how #2 and #3 serve that greater point. Documentation makes your system more understandable, which makes your team work faster. Code Quality makes your system adaptable to change, which makes your team work faster. For business people, it's all about maximizing the flow of value from the time an idea is pitched to the time the idea is delivered as working software.

Answer 2

One thing of definite value is that automate tests can be run continuously; such as every hour on a rebuild or similar. Doing this forces any bugs or regressions out into the open rapidly within hours or days of a programmer working on the offending code, this makes context switching much easier. The second benefit to continuous testing is that it forces you to keep you tests in a working state; nothing is more tedious than spending the first week of a test cycle fixing all of the out of date tests. If you can automate them, you can run them any time & by running them regularly you can catch bugs in your tests or code quickly.

Answer 3

Test Expense

Once an automatic test is written, it can be run by a computer at the cost of a few joules. The equivalent manual test requires a person on payroll working down a list of instructions.

Test Reliability

The computer can be trusted to faithfully execute the same test procedure, every time. The human is apt to make mistakes and get lazy.

The computer's testing failure modes are also much more readily apparent - it crashed (test reports stop appearing), it had a bit error that caused a false test result (run a deterministic test again, and the result differs). If a human misses a step and checks off the "OK", how can we tell?

Test Durability

An automated test has to be a concrete artifact (e.g. a piece of code) in order to run, and is naturally included with the other software development artifacts - the source repository. A manual test may be developed on a sheet of note paper by a tester, and never formalized. The business is more likely to need processes in place to ensure that doesn't happen.

Test Value

The computer can be programmed to output test results in a consistent, easily analyzed form. The person is either doing data entry to generate the same, or is recording free-form notes that require an analyst, developer, or manager to digest.

Answer 4

Mostly (depending on your test coverage) bug-free code, and I would say that one of the biggest arguments is when you say to your manager that you can write a test for a discovered bug, ensuring you will always know in the future if that bug comes back :)

My opinion is that unit / integration tests are most important, while if you apply some UI pattern like MVC it is enough for most projects. I usually test all actions on my controllers / presenters, and leave databinding to the Views.

Of course, automated testing does not substitute good old point and click adventuring around your application trying to figure out wildest things your user could do.

There is also a point of Continuous Integration.

One more thing - one must strive that code quality leads to product quality, business value, and maintainability - otherwise there is no point in doing it.

Answer 5

I think you should lead with the magic points of "lower Cost" and "more Features/unit time" / smaller cycle-time.

However before making a case, I'd advise to reflect on your situation. Your question led me to write a blog post on the potential cons of automated testing.

Answer 6

Ease of refactoring is a big factor here. Having good coverage by nice and READABLE (!!!) unit test, you can refactor your system without being nervous about compromising existing functionality.

Answer 7

You have to sell the concept – you need to avoid telling them it will improve the code. If they have any investment in the code that will immediately put them anti you/auto testing. If they are any good they will also understand GIGO so will not understand why you think it doesn't apply.

I'd leave selling it as documentation aspect out as well, things like Fitnesse can do it well, but until they experienced it might be hard to visualise.

Areas I think that might have luck selling it on

1. Unit Tests can take the place of many developer harnesses – where you create application just to get at area to debug/test without going through all the login/menus.

2. Tests allow you to set-up and easily repeat problems situations – without spending lots of time setting up test data (especially using a decent mocking system)

3. As you build up suites of BDD & UI tests – you get a much quicker response if there are simple breaks than waiting for next time the tester looks at it

4. BDD & UI tests can avoid you repeatedly pressing buttons to check all aspects that might have been affected by your change and save you having to remember every areas.

5. Automatic builds often highlight when someone has forgotten to check in code

6. Tests help you avoid bugs reappearing.

7. Unit Tests and decent mocking will mean less inter-linked code and will be easier to resolved

Remember you are trying to sell it, not convert them to a religion – so accept small steps and try not to get them anti-you. It will also take time for them to adjust and learn to write good tests.

Answer 8

Someone must beleive there is a problem before they will accept a proposed solution to that problem.

Automated tests can save bug-fixing costs, so if your colleagues don't beleive that bug-fixing costs are sizeable or excessive, they will be hard to convince. If those costs are high or excessive, but people do not believe they are, you might first have to get some convincing data about those costs.

Answer 9

What businesses love is increasing value and lowering cost. You have to explain how automated testing will increase value since it does add an extra cost.

If your code is modular it will be possible to re-use it. Which means that the tests do not have to written all over again and you can just work on top of that existing code.

If there are legacy projects, automated testing makes it much easier to refactor. The technical debt has to be paid down at some point.

The documentation argument you provide isn't very good. The difference between keeping tests up to date and documentation up to date is just habit.

Answer 10

"What I'm looking for help with is articulating its value to a team of programmers, analysts, managers and testers. By automated tests, I don't think I need to make a distinction between unit tests (e.g. JUnit), BDD (e.g. JBehave, Fitness) and UI (Selenium , Watir) because I think they all provide similar value (but feel free to write an answer that disagrees :))"

OK I'll take that challenge ;)

I mostly work with programmers and QA and my tools are ruby, rails, testunit, rspec, jasmine and selenium.

The BDD/TDD tools of rspec and testunit are part of programming. You don't break them out and talk about them separately to management, you don't put them off due to lack of time, you include them in all your time estimates. If really pushed asked how much time people have for you to explain computer science and programming to them. I don't use these tests for the front end

GUI/UI/Jasmine/Selenium. These are different. I have these done by QA folks who have programmer backgrounds. We make sure the tests are written to be as robust as possible and based on content not layout. The (possibly new) cost of this should be explained as a shifted cost. Instead of paying with broken software, lost customers and expensive fixes later, you pay a lot less (relatively) now with a few simple practices.

Answer 11

I think the key is to talk about specific categories of tests you will create, not 'automated testing' as a whole. The latter can be a bit nebulous and worrying, and it's too easy to come up with examples of where it would be waste of time.

I always recommend splitting your tests up into 4 groups (more detail here). Stick with me here, I'll get to how this helps you sell testing to others in a moment.

1. Tests of your core functionality. Ie, for a website monitoring tool this would be tests of alerts which should fire for websites you're monitoring. These tests make sure this stuff never breaks.
2. Smoke tests of your whole application. For example, using Selenium to navigate all links/buttons in a web app and make sure there are no errors from the server. These tests avoid you wasting testers time with obviously broken builds.
3. Tests of any fragile code. Ie, for that old module no one ever wants to touch, or the complex piece of code which seems to always have bugs in it.
4. Tests which devs wanted to write to support their work. Because sometimes tests are useful when you're writing something, but don't fall into the categories above.

By splitting your tests out into these categories you can now have a different discussion. Take the first three groups (the forth is at individuals discretion anyway) and ask if people think tests for those pieces of code would be worthwhile? If you can't get agreement, maybe you don't include those tests for now. If you can, ie if people agree that tests around core functionality which get executed on each commit are useful, then start adding those.

The other group which can be useful is tests which are hard or time consuming to do manually. There should be a pretty easy to explain benefit here in terms of saving manual testing time, or getting things tested which get skipped because of a lack of time.