How do you do continuous delivery in an unstable environment?

Question

We're Scrum teams building microservices. Our GitHub repositories are single-branch, each of us integrates his/her code into master several times a day, with no feature branches. Our Jenkins pipelines compile the code, run automatic tests, supply the code to other services for static code scans, and deploy our software across a CloudFoundry landscape for further testing. If all the steps succeed, our pipelines automatically deploy the software into production spaces on Amazon Web services. We live Uncle Bob's Clean Code, write reliable unit tests with > 90% mutation coverage, and honor Jez Humble's ideas from Continuous Delivery. We think we are doing everything right.

It doesn't work.

99% of our builds fail on their way through the pipelines. In many weeks our velocity is nearly 0.

Now the first impulse is to say we'd need to code cleaner, test more, stop pushing into red pipelines, roll back faster, perform root cause and post mortem analyses, and so on. But we've been there and done all that, chomped through our code, practices, and culture. We've streamlined and upskilled our own teams as best as we could.

The problem is that the builds fail for reasons that our teams have no control over: the Jenkins servers, the Maven Nexus, the npm registry, the code scanning services, the Cloud Foundry landscape, all of these are maintained by other teams in our company. Individually, all of the involved 15 tools and teams are nice, and suffer only sporadic outages that might block a pipeline from minutes to a handful of days max. But in combination, the failure probabilities sum up to a nearly impenetratable wall of random failure.

What are strategies to improve this situation?

Answer 1

I find it hard to believe that "only sporadic outages" of few dependent services would result in 99% builds failing. Maybe you need some detailed analysis on which services are worst offenders and either communicate with their owners to improve their stability or replaces them with more stable options.

Quick math. If each service has x% chance of failing and your build uses N services in sequence, then the probability of random build failure is : 1-(1-x)^N. So for failure rate of 1% and 5 services, it is 1-(1-0.01)^5 = 0.049, which is roughly 5%.

Another idea is to separate "necessary for deployment" dependencies from dependencies that provide useful information, but do not prevent deployment. Static code analysis comes to mind. You don't need it for deployment, but it is nice information. Instead of running it as part of your deployment pipeline, run it asynchronously in separate pipeline, that provides reports to developers.

Next idea that comes to mind is, that instead of single pipeline, you could have "phased" pipeline. Inspiration for this comes from book A Practical Approach to Large-Scale Agile Development, that Jez Huble likes to reference. There, they describe a build pipeline, that has multiple phases. Each able to run on it's own and which can be re-started if it fails, without need to re-start all that came before it.

Answer 2

Technical answer

The exact concept that is needed here is fault tolerance. You need a pipeline that is resilient to failure. This is an open-ended engineering problem, of course.

The most obvious brute force solution is redundancy, i.e. have redundant nodes in the pipeline so that even if something fails there is another server that can take over.

Another important but often neglected aspect to consider is the recovery mode. If there is a failure, does the pipeline pick up where it left off, or does it always start again from the beginning? Are build tasks idempotent and capable of being run repeatedly without issue, or do you have to trigger a full rollback and clean the system every time? This can make a huge difference in any process that has continuous problems.

Business answer

The other team needs help in getting visibility to the issues that are preventing them from deliveing a high quality, stable pipeline. Help them by gathering metrics of downtime and estimating the actual cost to the business. This will help them make a case with their own management to get additional resource (staff, hardware, training, or third party services) to meet the business' needs. You can also help with measuring success by coordinating and agreeing on an internal SLA and tracking their ability to meet it.

Needless to say, do not be adversarial about it, as this is often counterproductive.

Answer 3

Abandon the flaky network services, and do stuff locally.

You can unit test stuff on your own PC, and you can do it before you check in buggy code. You don't need a Maven repository if the third-party code is all in a repository that your team has control over. You don't need Jenkins to build source, instead fire up your IDE, and do a Clean and Build. If it doesn't absolutely need to be "in the cloud", do it locally within your team, and keep control over it.

Added bonus: In five years time, when a customer comes back and asks for an update to a product that you thought was long retired, you don't want to find that you have no way of building the thing anymore. Relying on a mess of servers maintained by multiple other people is asking for trouble. They will all have been retired, or moved on to newer versions that aren't compatible with your old project. It's much better if everything you need is configured locally, so you can check it all out and build the whole lot from scratch according to your documented procedures.