How short/stable should tests/build be for working CI/CD pipeline?

Question

(Please note that this question is linked to this: How can Continuous Delivery work in practice? - but it asks a more specific question regarding time and stability).

CI/CD, no manual QA, super quick releases are the new buzzwords and I have read about this topic many times, but it sounds somehow unrealistic for cases when automated tests are hard to pull out. Imagine a complex, UI interaction based software like a video editing software. Or one that has a huge backend, huge frontend with complex account setup, different kinds of authentication, etc. etc. Or if animations are a huge part of your frontend project.

For these big applications tests can run for hours (to give you perspective; in one of our projects full UI tests ran for 16 hours on just one platform; covering only 50% of manual test cases); also UI tests are notoriously brittle (even with very heavy investment on stabilizing them) and you absolutely don't want your release to be reverted after 15 hours because of an unreproducible random UI test bug.

So, the question is: how short and stable should the build/testing should be to be able to pull out a working CI/CD pipeline?

Answer 1

Typically you have multiple pipelines - you run "quick n dirty" unit tests as devs commit changes. When they say they're ready to merge their changes into a testing branch, you run a more "slow n steady" integration tests (and any other analysis tools you like). If these pass, you merge the testing branch onto some QA branch where you can generate deliveries from.

This 3-stage pipeline gives you all you need: quick verification, plus slow quality. You can't skip the quality checks no matter what the buzzword du jour says, internet opinion after all, is usually wrong.

In short - do what matters for you, if you want quality releases then you have to take the time to ensure releases are good.

Answer 2

I'd separate test stability from test duration, they matter in different manners.

Test stability is critical if the test is executed rarely and there aren't enough datapoints to actually measure the stability. Checken and egg problem.

But if enough execution datapoints exists to allow an estimation of the test stability then it can be incorporated in the configuration of the CI/CD pipeline such that the effect of occasional glitches is reduced, maybe even significantly. For example automatic (or manual, after human analysis) test retry before reverting the release in the case you mentioned, if the respective test is known to randomly fail fairly often. The number and timing of retries would depend on the estimated rate of random failures.

A smart CI/CD system can include monitoring the quality verification failure rates and even help in identifying the intermittent issues, or at least the most damaging ones.

As for the test duration - that's not really relevant. In the vast majority of cases splitting a very long test into multiple shorter tests that can be executed in parallel is a typical technique to shorten a CI/CD pipeline execution (for a price, of course).