What are the disadvantages of automated testing?

Question

There are a number of questions on this site that give plenty of information about the benefits that can be gained from automated testing. But I didn't see anything that represented the other side of the coin: what are the disadvantages? Everything in life is a tradeoff and there are no silver bullets, so surely there must be some valid reasons not to do automated testing. What are they?

Here's a few that I've come up with:

• Requires more initial developer time for a given feature
• Requires a higher skill level of team members
• Increase tooling needs (test runners, frameworks, etc.)
• Complex analysis required when a failed test in encountered - is this test obsolete due to my change or is it telling me I made a mistake?

Edit
I should say that I am a huge proponent of automated testing, and I'm not looking to be convinced to do it. I'm looking to understand what the disadvantages are so when I go to my company to make a case for it I don't look like I'm throwing around the next imaginary silver bullet.

Also, I'm explicity not looking for someone to dispute my examples above. I am taking as true that there must be some disadvantages (everything has trade-offs) and I want to understand what those are.

Answer 1

You pretty much nailed the most important ones. I have a few minor additions, plus the disadvantage of tests actually succeeding - when you don't really want them to (see below).

• Development time: With test-driven development this is already calculated in for unit tests, but you still need integration and system tests, which may need automation code as well. Code written once is usually tested on several later stages.

• Skill level: of course, the tools have to be supported. But it's not only your own team. In larger project you may have a separate testing team that writes tests for checking the interfaces between your team's product and other's. So many more people have to have more knowledge.

• Tooling needs: you're spot on there. Not much to add to this.

• Failed tests: This is the real bugger (for me anyways). There's a bunch of different reasons, each of which can be seen as a disadvantage. And the biggest disadvantage is the time required to decide which of these reasons actually applies to your failed test.

  • failed, because of an actual bug. (just for completeness, as this is of course advantageous)
  • failed, because your test code has been written with a traditional bug.
  • failed, because your test code has been written for an older version of your product and is no longer compatible
  • failed, because the requirements have changed and the tested behavior is now deemed 'correct'

• Non-failed tests: These are a disadvantage too and can be quite bad. It happens mostly, when you change things and comes close to what Adam answered. If you change something in your product's code, but the test doesn't account for it at all, then it gives you this "false sense of security".

  An important aspect of non-failed tests is that a change of requirements can lead earlier behavior to become invalid. If you have decent traceability, the requirement change should be able to be matched to your testcode and you know you can no longer trust that test. Of course, maintaining this traceability is yet another disadvantages. And if you don't, you end up with a test that does not fail, but actually verifies that your product works wrongly. Somewhere down the road this will hit you.. usually when/where you least expect it.

• Additional deployment costs: You do not just run unit-tests as a developer on your own machine. With automated tests, you want to execute them on commits from others at some central place to find out when someone broke your work. This is nice, but also needs to be set up and maintained.

Answer 2

Having just started trying automated tests in our team, the biggest disadvantage I've seen is that it's very difficult to apply to legacy code that wasn't designed with automated testing in mind. It would undoubtedly improve our code in the long term, but the level of refactoring needed for automated testing while retaining our sanity is a very high barrier for entry in the short term, meaning we're going to have to be very selective about introducing automated unit testing in order to meet our short term commitments. In other words, it's a lot harder to pay off the credit cards when you're already deep in technical debt.

Answer 3

Perhaps the most important disadvantage is ... tests are production code. Every test you write adds code to your codebase that needs to be maintained and supported. Failing to do so leads to tests you don't believe the results of, so you have no other choice. Don't get me wrong - I'm a big advocate of automated testing. But everything has a cost, and this is a big one.

Answer 4

I wouldn't say these are entirely applicable disadvantages, but the few I hit most are:

• Time taken to set up the test in a complex enterprise application.
• Handling old tests that fail incorrectly, in other words, the system has evolved and now the tests are wrong.
• False confidence from patchy or unknown test coverage.

Test coverage that is patchy can lead to a false sense of security. If you do a refactor and use the tests to prove it's validity, what has proven your tests are able to prove this?

The time it takes to create the test is sometimes a problem for us. Our automated testing doesn't just include unit tests, but also use case tests. These tend to be broader and require context.

Of course, my perspective is from an application that is older than it's unit tests.

Answer 5

I'd say the main problem with them is that they can provide a false sense of security. Just because you have unit tests it doesn't mean they're actually doing anything and that includes properly testing the requirements.

In addition automated tests can also include bugs themselves, so that brings the question over whether unit tests need testing themselves so not necessarily achieving anything.

Answer 6

Though automation testing has many advantages, it has its own disadvantages too. Some of the disadvantages are:

• Proficiency is required to write the automation test scripts.
• Debugging the test script is major issue. If any error is present in the test script, sometimes it may lead to deadly consequences.
• Test maintenance is costly in case of playback methods. Even though a minor change occurs in the GUI, the test script has to be rerecorded or replaced by a new test script.
• Maintenance of test data files is difficult, if the test script tests more screens.

Some of the above disadvantages often subtract from the benefit gained from the automated scripts. Though the automation testing has pros and corns, it is adapted widely all over the world.

Answer 7

I recently asked a question about tests in game development - this is BTW how I knew about this one. The answers there pointed some curious, specific disadvantages:

1. It is costly to do when your code should be highly coupled.
2. It is difficult to do when you have to be aware of the various hardware platforms, when you should analyze output to the user and the code result only makes sense in a broader context.
3. UI and UX testing is very hard.
4. And notably, automated tests can be more expensive and less effective than a bunch of very low-cost (or free) beta testers.

The 4th point makes me remember of some experience of mine. I worked on a very lean, XP-oriented, Scrum managed company where unit tests were highly recommended. However, in its path to a leaner, less bureaucratic style, the company just neglected the construction of a QA team - we had no testers. So frequently the customers found trivial bugs using some systems, even with test coverage of >95%. So I would add another point:

• Automated tests may make you feel that QA and testing are not important.

Also, I was thinking those days about documentation and cogitated a hypothesis that may be valid (to a lesser extend) to tests two. I just felt that code evolves so quickly that it is pretty hard to make documentation that follows such a velocity, so it is more valuable to spend time making code readable than writing heavy, easily outdated documentation. (Of course, this does not apply to APIs, but only to internal implementation.) The test suffers a bit from the same problem: it may be too slow to write when compared with the tested code. OTOH, it is a lesser problem because the tests warn they are outdated, while your documentation will stay silent as long as you do not reread it very, very carefully.

Finally, a problem I find sometimes: automated testing may depend upon tools, and those tools may be poorly written. I started a project using XUL some time ago and, man, this is just painful to write unit tests for such platform. I started another application using Objective-C, Cocoa and Xcode 3 and the testing model on it was basically a bunch of workarounds.

I have other experiences about disadvantages of automated testing, but most of them are listed in other answers. Nonetheless, I am a vehement advocate of automated testing. This saved an awful lot of work and headache and I always recommend it by default. I judge those disadvantages are just mere details when compared to the benefices of automated testing. (It is important to always proclaim your faith after you comment heresies to avoid the auto da fé.)

Answer 8

Two that aren't mentioned are:

• Length of clock time it may take to run a large test suite

I've been a part of automated QA efforts where it took half a day to run the tests, because the tests were slow. If you're not careful with writing your tests, your test suite could turn out this way too. This doesn't sound like a big deal until your now managing that time, "Oh, I just committed a fix, but it's going to take 4 hours to prove correctness".

• Frailty of some test writing methods

Some testing methods (such as automating the UI) seem to break every time you turn around. Especially painful if your script, say, hangs the testing process because it's waiting for a button to appear - but the button has been renamed.

These are both things you can work around, with good testing practice: find ways to keep your test suite fast (even if you have to do tricks like distributing test runs across machines/CPUs). Likewise, some care can be taken to try to avoid frail methods of writing tests.

Answer 9

I want to add one more, a false sense of security.

Beyond very small well defined problem sets, it is not possible to create comprehensive tests. There may and often will still be bugs in our software that the automated tests simply don't test for. When the automated tests pass we all too often assume there are no bugs in the code.

Answer 10

It is hard to convince the management/venture capitalist

• testautomation increases upfront cost.
• testautomation increases time to market.
• the benefit of testautomation comes in the mid- and logterm. fierce competition focuses more on shortterm benefits.

see Market Driven Unit Testing for details.

Answer 11

One of the main disadvantages can be overcome by using self-learning tests. In this situation the expected results are all stored as data and updateable with minimal review by test suite user in self-learning mode (show diffs between old expected result and new actual result - update expected if press y). This testsuite learning mode needs to be used with caution so buggy behaviour is not learned as being acceptable.