The ideal goal for my system is the ability to perform an arbitrary number of deployments per day to production while making sure each 'push' goes through the full test suite to make sure it holds up to our specifications/user stories.

This is a manageable challenge with Integration / Unit tests which usually are rock solid but fails to be reasonable when I try to test E2E tests which are notoriously flakey (and sometimes fail due to unknown circumstances).

I often use E2E testing to test full User stories using NightwatchJS. A typical test may start from a login and consisting of a sequence of complicated steps which are executed in order. I try my best to make sure all E2E tests are as small as possible (so when they fail other user stories are not skipped) but I am still subject to often random test failures which make running them as part of the deployment pipeline a no-go.

I therefore have a few questions about how other organizations who acheive multiple deployments per day run manage testing in their deployment pipeline:

  1. What rate of failure do you find for your E2E testing suite? How do you work as a team to improve it
  2. At what stage of the deployment do you run E2E tests: Automatically before merges to master? Manually? Does a small backend change result in a E2E suite run? How long does it extend your deployment pipeline by?
  3. Does an E2E test failure equal a deployment failure?
  4. In what environment do you run E2E tests, on local development machines? in QA servers? on shared development environments? on Staging?

I'm really interested in finding out at what point your E2E test suites run and what levels of flakiness you experience. Any input would be great!

Cheers Deepak

  • 1
    This shouldn't be on hold, its not a broad question and it asks for real world examples of where people run their E2E tests. – Deepak Jul 27 '18 at 14:23
  • 1
    Your question is too broad. To make it more suitable for this site, focus it more specifically on the real-world problem you are having, and make it less of a "share your experiences" question. – Robert Harvey Jul 27 '18 at 14:55
  • 2
    @Deepak I think it's a good question, but I think you would be unlikely to get a good answer. If your e2e tests take longer to run than you have time to run them in there's no solution. My answer would basically be, you can do lots of deployments, but not all to the same system, fix your tests they should run perfectly and buy more test servers. I could throw in some rough examples, but its not going to tell you anything you dont already know – Ewan Jul 27 '18 at 15:19
  • maybe pick the top thing causing your e2e tests to be flakey and ask for strategies on how to fix that – Ewan Jul 27 '18 at 15:22