Are end-to-end and integration tests worth it for non-mission critical stuff?

Question

It's well known that end-to-end and integration tests are costly. Of course if we develop applications where people might die if things go wrong it's a worthwhile investment. However in applications where errors are not the end of the world, would it not be cheaper to skip the E2E tests and integration tests altogether and instead craft a backup plan if something goes wrong? Like is a manual test of user stories + unit tests + using a statically typed language enough?

Like for example if a web store lost an order they could instead send the item for free + another item as an apology. The end user might end up even happier that way and the company overall saves money.

I guess my question is, in general how much does an integration test and E2E test cost and how much money does it save? Is there a way to do a risk/cost calculation for this?

Answer 1

It does not matter if you implement E2E and integration tests or not, you need a backup plan either way. Never expect a system to be bug-free just because it was tested.

Thus, in your cost estimation, you do not compare the cost for implementing E2E tests against the costs your backup plan estimates in case of a failure, you compare:

• Costs for doing E2E tests manually (several times, before each new release)

vs.

• Costs for building (and maintaining) automated E2E tests

In case you can use those E2E tests several times, there will usually be a number of test runs where the costs reach a break-even point. That should be the metric you apply when to want to plan ahead which E2E tests you will do manually, and which you will automate.

Note there may be some kinds of E2E tests which can be implemented easily, where the ROI is immediately clear, but there are also kinds of E2E tests where development and maintenance may be more expensive that doing them manually over a period of several years.

Answer 2

Perhaps counter intuitively, automated testing can actually reduce development time vs no testing. So it's a win win.

The idea is that the tests contribute on a number of levels

1. Force strict requirement gathering and specification

   This makes a huge impact on the speed of development. No going back asking for more detail, no misunderstandings, no unneeded features etc

2. Developers know when a feature is complete

   Most testing is done by developers during the writing of the code rather than testers checking a finished product. Automating this testing reduces this workload

3. Bugs introduced by new features instantly detected.

   These can easily cost you a sprint and require the rewriting of an entire feature if they go undetected.

4. Faster release cycle

   This means less code in flight, which means less merging, which means less work and less complexity for developers

Especially if you have a test framework setup, writing these tests takes less time than you save in these efficiencies.

Plus, you save on manual testing time, plus you get a better product at the end.

Answer 3

In my experience E2E testing, regardless of the criticality of the app, is always prudent. I always think in terms of worst case scenario, if things go pear shaped are you comfortable standing up in front of management and justifying your approach? If not, then you need to change your approach. Many organisations minimise the importance of and resources allocated to testing, but rest assured that when things go wrong everyone is looking for someone to blame and if you made the decision to limit testing or gave that advice, then you're the one in a firing line.

Software development all too often requires an eye on organisational politics.

Answer 4

My answer? Maybe, probably not.

EOE tests are good when they are very simple. If you are planning to cover basic scenarios, you can manage to gain some advantage with EOE tests. But if you have a really complex and big application (mission critical or not), this EOE tests will be expensive to maintain and you need to know your scenario to valuate if worth it.

Some years ago the Google Testing Blog discuss this subject. I can only agree with the author. A good test needs to be fast, reliable and isolate failures, features that the EOE tests are not capable to deliver to you.

I worked on an application that have more than 12 hours of end-to-end tests covering a lot of scenarios. Eventually we managed to distribute this tests on different machines, controlling the start, execution and ending of the tests, collecting and merging the results. The tested application was a monolith application (what it is easier to put up and running to test) and was nightmare to maintain the tests.

Most part of the time we was maintaining the tests instead of catching bugs from their results. Discover the origin of a bug on a end-to-end test takes a lot of time. We also dealt with a lot of "false-negative" tests and few time to understand the problem and correct it: Java Applet loading problems, expected element not found on the page (plus other problems about the automation speed), maintain query code that are just used on the database memory test (because the original query use database specific code), etc.

All of this need people to maintain up and running. On the end we starting to delete some EOE tests and replace them with many unit/integration tests.

So, my conservative advice is use the testing pyramid from Google:

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape.

Answer 5

"It's well known that end-to-end and integration tests are costly."

I think I disagree with this assertion.

Firstly, E2E tests are what matters to end-users and can be the most time-effective/lowest cost options for testing complex systems. For instance, when someone buys a car most people do not pull it to pieces and start testing the carb, gearbox, wheels in isolation. Instead, they take it for a test drive.

Secondly, in terms of tooling, E2E doesn't tend to slow down the internal evolution of the product and lasts longer. If you think about it, the actual functional surface of most products rarely changes that much, while internally it can be subject to all sorts of developments. As a result, once the test tooling is up and running, it usually lasts extremely well. As an example, if we go back to the car analogy. The same "take it for a drive" test case would pretty much work on Ford Model T as on a Tesla. As would investments in rolling roads, wind tunnels, leak testing setups etc. How many of the internal component tests would have had such a good ROI over their life spans?

Where E2E testing does tend to be more expensive/inappropriate though is in the initial setup and if it is used to try and test everything. Pragmatically, I think the best way to avoid this trap is to priorities automating the testing of things that:

1. Are easy to automate and unlikely to need much maintenance to keep running.
2. Consume the most time to apply consistent, adequate, manual testing processes to.
3. Risk making you or your boss look like idiots if the product is released with it broken.

Use whatever form of testing including E2E you think is appropriate. Focus on those though.

Answer 6

You can't really compare the cost of integration tests to the cost of a best case scenario where a bug only affects a single order. A logical bug would be just as likely to affect a large number of orders. Say a bug means that no payments are captured - this could have disastrous effects for any business.

You should ask what is the worst case bug which realistically could end up in production due to the lack of E2E-testing. And remember Murphys law.

Answer 7

I assume that this question is about enterprise web applications.

My recommendation for medium-critical stuff:

• Perform automated testing for your backend APIs, making sure that the backend works as expected. Ideally, tests should be written by the developers while implementing an API.
• Do not care that much about automated UI tests, that is, do the frontend testing manually.

I think that most tests should be on the API level or component level. In many cases, I do not care about unit tests that are only executing some internal functions.