Recently, I was charged with making about 9000 Selenium tests start running in CI/CD nightly. These tests had built up over about 8 years and had up until then been run in an ad hoc way. It was important that failures in the CI/CD be genuine failures, which meant that you could run the test as many times as you liked and it would continue to fail.
Anyway, my immediate observation was that lots of these tests unreliable in the sense of would sometimes fail for all sorts of spurious reasons, test issues, like not waiting for something to be present and so only passing if that thing was only coincidentally present. Sometimes one test would change the state of the application and another test might rely on that being/not being the case, etc. It was basically arbitrarily complicated reasons a test could be flaky.
My 'resolution' was, I introduced a lot of retry logic and added mechanisms by which the state of the application got reset between retries. To handle the overhead all these retries, etc., I introduced a huge amount of parallelism to the testing process, split across multiple deployments of the application to mitigate other limitations and in this way managed to make the tests run in a time frame that was acceptable to the business.
Everyone is happy now, in as much as the original task was achieved and the CI/CD is providing meaningful value and identifying genuine issues introduced.
I'm not so happy, because I want the tests themselves to be reliable, because unreliability is still harmful with respect to running these tests locally - and I want people to be able to identify if their changes will cause a failure pre merging, and also because the tests would run around 8x faster without the retry overhead.
What is a process I can follow to actually iteratively achieve this? It's hard to present a genuine business justification. My point about running the tests locally isn't much of an argument, because no one does that anyway (because they're unreliable, so it is a bit circular). And, as I say, the business is happy with the time, and the design of things is such that you can always throw more resources at the tests to make them faster.