9

It's well known that end-to-end and integration tests are costly. Of course if we develop applications where people might die if things go wrong it's a worthwhile investment. However in applications where errors are not the end of the world, would it not be cheaper to skip the E2E tests and integration tests altogether and instead craft a backup plan if something goes wrong? Like is a manual test of user stories + unit tests + using a statically typed language enough?

Like for example if a web store lost an order they could instead send the item for free + another item as an apology. The end user might end up even happier that way and the company overall saves money.

I guess my question is, in general how much does an integration test and E2E test cost and how much money does it save? Is there a way to do a risk/cost calculation for this?

  • 4
    Is there a way to do a risk/cost calculation for this? Other than actually doing both and then comparing, no. – Robbie Dee Mar 13 at 8:43
  • 4
    You must consider the ROI of everything in the development process. Are unit tests worth it? Is manual testing worth it? Is code quality worth it? Is creating the software in the first place worth it? Those are important business questions. Which of course cannot be answered in general. And they are more about business administration than about software engineering. – Christian Hackl Mar 13 at 8:59
  • What do you think the business impact is if a web store like Amazon lost a few hours or orders? They can try to re-send the items at no cost, but what would it do to their reputation? – Bart van Ingen Schenau Mar 13 at 10:23
  • @BartvanIngenSchenau I think a massive company like Amazon can afford integration tests and E2E. It's easy to see few hours of orders costing them millions. But I wonder if for smaller companies the cost to reputation is less than the cost of the tests themselves. Especially since turning unhappy customers into happy ones can be extremely valuable means that it might not even be a cost to begin with. I just don't have any experience to draw conclusions on, hence why i'm asking. – Marc Mar 13 at 10:30
12

It does not matter if you implement E2E and integration tests or not, you need a backup plan either way. Never expect a system to be bug-free just because it was tested.

Thus, in your cost estimation, you do not compare the cost for implementing E2E tests against the costs your backup plan estimates in case of a failure, you compare:

  • Costs for doing E2E tests manually (several times, before each new release)

vs.

  • Costs for building (and maintaining) automated E2E tests

In case you can use those E2E tests several times, there will usually be a number of test runs where the costs reach a break-even point. That should be the metric you apply when to want to plan ahead which E2E tests you will do manually, and which you will automate.

Note there may be some kinds of E2E tests which can be implemented easily, where the ROI is immediately clear, but there are also kinds of E2E tests where development and maintenance may be more expensive that doing them manually over a period of several years.

  • Thanks, this is a great answer. What are examples of E2E tests that are easy to implement but lead to more development and maintenance down the line? – Marc Mar 14 at 3:13
  • 2
    @Marc: I guess you misunderstood my last sentence, I was talking about different tests: the ones which are easy to implement/maintain, and the ones which are not. – Doc Brown Mar 14 at 4:26
  • Correct, the edited version makes it clearer. – Marc Mar 14 at 4:29
  • @Marc: To my experience, tests through user interfaces (especially complex ones) are often a candidate where test automation is less cost-effective than other tests - but it depends a lot on the category of software, the available tools, and the specifc technologies involved. – Doc Brown Mar 14 at 4:32
7

Perhaps counter intuitively, automated testing can actually reduce development time vs no testing. So it's a win win.

The idea is that the tests contribute on a number of levels

  1. Force strict requirement gathering and specification

    This makes a huge impact on the speed of development. No going back asking for more detail, no misunderstandings, no unneeded features etc

  2. Developers know when a feature is complete

    Most testing is done by developers during the writing of the code rather than testers checking a finished product. Automating this testing reduces this workload

  3. Bugs introduced by new features instantly detected.

    These can easily cost you a sprint and require the rewriting of an entire feature if they go undetected.

  4. Faster release cycle

    This means less code in flight, which means less merging, which means less work and less complexity for developers

Especially if you have a test framework setup, writing these tests takes less time than you save in these efficiencies.

Plus, you save on manual testing time, plus you get a better product at the end.

  • To me, this answer stands or falls depending on whether the OP is talking about integration tests over and above unit tests. If there already are unit tests, then the answer would seem to be speculative at best. If there are no unit tests, then naturally - some automated testing is better than none at all. – Robbie Dee Mar 13 at 13:01
  • Yeah, I assume that we have unit tests in place – Marc Mar 14 at 3:05
1

My answer? Maybe, probably not.

EOE tests are good when they are very simple. If you are planning to cover basic scenarios, you can manage to gain some advantage with EOE tests. But if you have a really complex and big application (mission critical or not), this EOE tests will be expensive to maintain and you need to know your scenario to valuate if worth it.

Some years ago the Google Testing Blog discuss this subject. I can only agree with the author. A good test needs to be fast, reliable and isolate failures, features that the EOE tests are not capable to deliver to you.

I worked on an application that have more than 12 hours of end-to-end tests covering a lot of scenarios. Eventually we managed to distribute this tests on different machines, controlling the start, execution and ending of the tests, collecting and merging the results. The tested application was a monolith application (what it is easier to put up and running to test) and was nightmare to maintain the tests.

Most part of the time we was maintaining the tests instead of catching bugs from their results. Discover the origin of a bug on a end-to-end test takes a lot of time. We also dealt with a lot of "false-negative" tests and few time to understand the problem and correct it: Java Applet loading problems, expected element not found on the page (plus other problems about the automation speed), maintain query code that are just used on the database memory test (because the original query use database specific code), etc.

All of this need people to maintain up and running. On the end we starting to delete some EOE tests and replace them with many unit/integration tests.

So, my conservative advice is use the testing pyramid from Google:

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape.

0

In my experience E2E testing, regardless of the criticality of the app, is always prudent. I always think in terms of worst case scenario, if things go pear shaped are you comfortable standing up in front of management and justifying your approach? If not, then you need to change your approach. Many organisations minimise the importance of and resources allocated to testing, but rest assured that when things go wrong everyone is looking for someone to blame and if you made the decision to limit testing or gave that advice, then you're the one in a firing line.

Software development all too often requires an eye on organisational politics.

0

"It's well known that end-to-end and integration tests are costly."

I think I disagree with this assertion.

Firstly, E2E tests are what matters to end-users and can be the most time-effective/lowest cost options for testing complex systems. For instance, when someone buys a car most people do not pull it to pieces and start testing the carb, gearbox, wheels in isolation. Instead, they take it for a test drive.

Secondly, in terms of tooling, E2E doesn't tend to slow down the internal evolution of the product and lasts longer. If you think about it, the actual functional surface of most products rarely changes that much, while internally it can be subject to all sorts of developments. As a result, once the test tooling is up and running, it usually lasts extremely well. As an example, if we go back to the car analogy. The same "take it for a drive" test case would pretty much work on Ford Model T as on a Tesla. As would investments in rolling roads, wind tunnels, leak testing setups etc. How many of the internal component tests would have had such a good ROI over their life spans?

Where E2E testing does tend to be more expensive/inappropriate though is in the initial setup and if it is used to try and test everything. Pragmatically, I think the best way to avoid this trap is to priorities automating the testing of things that:

  1. Are easy to automate and unlikely to need much maintenance to keep running.
  2. Consume the most time to apply consistent, adequate, manual testing processes to.
  3. Risk making you or your boss look like idiots if the product is released with it broken.

Use whatever form of testing including E2E you think is appropriate. Focus on those though.

0

You can't really compare the cost of integration tests to the cost of a best case scenario where a bug only affects a single order. A logical bug would be just as likely to affect a large number of orders. Say a bug means that no payments are captured - this could have disastrous effects for any business.

You should ask what is the worst case bug which realistically could end up in production due to the lack of E2E-testing. And remember Murphys law.

0

I assume that this question is about enterprise web applications.

My recommendation for medium-critical stuff:

  • Perform automated testing for your backend APIs, making sure that the backend works as expected. These tests should be written by the developers while implementing an API.
  • Do not care about automated UI tests, that is, do the frontend testing manually.

I think that most tests should be on the API level or component level. I do not care about unit tests that are only executing some internal functions.

protected by gnat Mar 14 at 12:50

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.