The ideal goal for my system is the ability to perform an arbitrary number of deployments per day to production while making sure each 'push' goes through the full test suite to make sure it holds up to our specifications/user stories.
This is a manageable challenge with Integration / Unit tests which usually are rock solid but fails to be reasonable when I try to test E2E tests which are notoriously flakey (and sometimes fail due to unknown circumstances).
I often use E2E testing to test full User stories using NightwatchJS. A typical test may start from a login and consisting of a sequence of complicated steps which are executed in order. I try my best to make sure all E2E tests are as small as possible (so when they fail other user stories are not skipped) but I am still subject to often random test failures which make running them as part of the deployment pipeline a no-go.
I therefore have a few questions about how other organizations who acheive multiple deployments per day run manage testing in their deployment pipeline:
- What rate of failure do you find for your E2E testing suite? How do you work as a team to improve it
- At what stage of the deployment do you run E2E tests: Automatically before merges to master? Manually? Does a small backend change result in a E2E suite run? How long does it extend your deployment pipeline by?
- Does an E2E test failure equal a deployment failure?
- In what environment do you run E2E tests, on local development machines? in QA servers? on shared development environments? on Staging?
I'm really interested in finding out at what point your E2E test suites run and what levels of flakiness you experience. Any input would be great!