8

The organisation I work for has recently employed a testing officer to run manual tests, but when I asked about being given time as a developer to write unit tests the response was that the manual testing would provide a bigger bang for the buck. That is something that feels wrong to me and I'm looking for a means to rate manual and automated testing against each other that is more scientific than a gut-feeling. I'm not saying there is no place for manual testing - but automated testing would seem, to me at least, to remove some of the more repetitive and boring tasks. We have a build server that runs some unit tests and some selenium tests - so its not like the idea of automated testing is a no-go, but just seen as a lower return on investment.

I can understand that having someone perform a full end to end test of a system tests the final product and ultimately that's all the user cares about, but it is slow going and very repetitive. Manual regression testing means repeating all the previous tests and confirming nothing has changed and if there's 4 paths through a process then that's 4 manual tests that could well take 5 minutes each.

So are there any verifiable facts and figures I can use to make the case for budgeting time for automated testing? For that matter what are the disadvantages to automated testing beyond those in the link?

Improve this question
2
  • 1
    Vote to close - there is not enough information (and probably never can be) about the details of the organization and software to be tested to provide a meaningful answer. Any meaningful answer will likely be too specific for the general community
    – mattnz
    Feb 11, 2013 at 21:45
  • @mattnz I appreciate that a full answer would require knowledge of the organisation, the culture, finances etc. but conceptually I would like to know if there is any objective arguments I can use to make the case for more automated testing.
    – Simon Martin
    Feb 12, 2013 at 9:38

3 Answers 3

Reset to default
14

I'd be careful at all when thinking about "manual" testing vs "automated" testing in a return-on-investment (ROI) context. It's a form of fallacious thinking when discussing the value of automation (see the "Time Machine" section of that link).

The Time Machine fallacy summed up is this: the only way you can truly determine the ROI of a testing approach is by having a time machine where you can visit the future and see what happened. Since you (most likely) can't do this, ROI estimations should be seen as a heuristic at best, and not some hard rule. Use current information to make an informed decision about a given testing effort is useful to your team or not.

Instead, when it comes to testing, think about which areas of your product need the most attention, and where the largest risks may be. Perhaps it's been identified that end-to-end integration testing might find the most issues with your app. Even if unit testing brings benefits, those benefits may not be worth the associated costs (getting developer buy in, writing and running tests, etc). Likewise, try thinking about benefits/costs instead of advantages/disadvantages for testing.

Improve this answer
3
  • 3
    There's a +1 waiting for you if can put a summary of the point you link to in your answer...
    – vaughandroid
    Feb 11, 2013 at 17:18
  • The +1 is yours! :)
    – vaughandroid
    Feb 11, 2013 at 17:51
  • 4
    +1 from me, too, though I guess there is a good chance that the one who told the OP that "manual testing provides the bigger bang for the buck" did some fallacious thinking, too.
    – Doc Brown
    Feb 11, 2013 at 17:56
10

First, you should clearly differentiate unit tests and other automated tests. Those are two different things, with different aims.

Unit tests are tests to validate very small portions of your code. They are typically white box tests, written by a developer who knows the source code "under test". In TDD, they are written in a test-first manner, by a dev who is going to write the source code after he wrote the test. Those kind of tests cannot be transferred to a "test officer" by their nature, and if your organization is not completly misguided, you should not have to ask your boss if and when you are allowed to write them - just as you should not have to ask your boss if and when you are allowed to write comments into your source code.

Integration tests, acceptance tests and other black-box tests are a different thing. If you have a testing officer who is going to apply those tests just manually, and he has to repeat the same kind of tests over and over again, the need for automation should be requested by him, not by you (of course, you should make transparent to him what kind of tests you can automate, and which not). He should tell you for which kind of tests he thinks that automation will make sense, and for which one not. Perhaps he is going to do the automation himself, using some regression testing tool, or he will ask your help for creating some tools for him.

Of course, when you think if he is not experienced or smart enough to come to that conclusion by himself, you should talk to him (and if that does not help, perhaps to your boss). However, IMHO you should do this only when you get the impression that your testing officer is a bottleneck in your production because of some missing test automation.

Improve this answer
3

Everyone always wants external examples and studies, but these aren't very effective in convincing management. If manual testing is truly a bottleneck at your company, then you will have all the internal examples you need:

A bug accidentally goes out the door. You state that it used to work and request it get added to the regression tests. Manual tester states it will make regression testing take too long. You offer to make an automated test for it. Manual tester welcomes the idea because he has been feeling overwhelmed and would like more time to focus on the more "interesting" test cases. With the voices of angry customers still ringing in their ears, management starts to see your point about test automation.

If meetings like this aren't happening at your company, no amount of external studies will make your case.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.