We're Scrum teams building microservices. Our GitHub repositories are single-branch, each of us integrates his/her code into master several times a day, with no feature branches. Our Jenkins pipelines compile the code, run automatic tests, supply the code to other services for static code scans, and deploy our software across a CloudFoundry landscape for further testing. If all the steps succeed, our pipelines automatically deploy the software into production spaces on Amazon Web services. We live Uncle Bob's Clean Code, write reliable unit tests with > 90% mutation coverage, and honor Jez Humble's ideas from Continuous Delivery. We think we are doing everything right.
It doesn't work.
99% of our builds fail on their way through the pipelines. In many weeks our velocity is nearly 0.
Now the first impulse is to say we'd need to code cleaner, test more, stop pushing into red pipelines, roll back faster, perform root cause and post mortem analyses, and so on. But we've been there and done all that, chomped through our code, practices, and culture. We've streamlined and upskilled our own teams as best as we could.
The problem is that the builds fail for reasons that our teams have no control over: the Jenkins servers, the Maven Nexus, the npm registry, the code scanning services, the Cloud Foundry landscape, all of these are maintained by other teams in our company. Individually, all of the involved 15 tools and teams are nice, and suffer only sporadic outages that might block a pipeline from minutes to a handful of days max. But in combination, the failure probabilities sum up to a nearly impenetratable wall of random failure.
What are strategies to improve this situation?