How to rollback rejected features by QA in a Continuous Delivery scenario?

Question

As I am looking more and more into Continuous Delivery for our organisation, I struggle to understand how we can effectively incorporate manual QA testing before a release is pushed out into production.

I have found multiple approaches, such as the branch-per-feature one explained by Adam Dymitruk or git-flow and the likes, which all seem to include a lot of manual work to get things setup each sprint or perform rollback operations.

I'll try to explain my issue by given an example.

Most CD documents outline the "happy" path flow as follows:

develop -> CI build -> deploy to QA -> test on QA -> promote to Production

This is easy enough to understand and works fine if there are no issues. But let's assume the following workflow which requires manual testing before a deploy to production is authorized: developer 1 works on feature A, developer 2 works on feature B. I use the name features, but I don't necessarily mean this in the broad range of a feature which could be contained by using feature toggles. Just some work.

developer 1 finishes his work, merges everything into master - or the equivalent release branch -, triggers the build which is deployed into QA. QA starts testing. During QA tests, developer 2 finishes his work, merges into master, triggers the build. QA then rejects feature A. Thus, the release from developer 1 cannot continue. At this point the entire process halts. As long as feature A is not ready, feature B cannot be tested because it also contains the rejected code from feature A.

And this is the part which I don't really understand how to fix in an easy, automated way. How can we effectively rollback feature A without complex manual interventions? How do people do this? Surely this must be an issue everyone faces, but none of the books or schematics I've read so far seem to really address this issue.

Taking it one step further. GitHub flow seems to suggest to deploy the feature branch into production, and rollback if a production issue occurs. That might work in a single developer single tester scenario, but what happens if multiple features are signed of by multiple testers at the same time?

developer 1 starts his feature, branches of master, starts working, pushes into master, QA 1 tests. At the same time, developer 2 went through the same process and QA 2 tests his feature build. Given the schematic, feature A does not contain feature B and feature B does not contain feature A. QA 1 signs off, feature A goes into production. QA 2 signs off, feature B goes into production. But feature A is lost. It's in master, yes, after the deploy it had been merged, but it won't get back into production until feature C gets deployed.

How would GitHub flow effectively work in larger teams? Has anyone some practical examples on this?

Answer 1

When a code change A is rejected, don't think necessarily in terms of "rollback feature A by undoing the merge using SCCS". Think in terms of "adding a new change to the code which fixes the defects found by QA".

Depending on the issue, this can mean:

• fixing a bug immediately, but leaving the code of A in master

• disabling the functionality added by A (by deactivating a feature entry point, maybe by utilizing a "feature toggle"), until the issues can be fixed, but still leaving the related code in master

• adding changes to the code which effectively remove the code changes of A from the master - but in a manner this does not interfer with later changes like B

The third bullet point can often, but not necessarily, mean to let the developer (and not the QA team by some magic automatism) use the SCCS and revert the code changes of A. If this rollback shows no collision with B, fine, if there is a collision, there might be some manual work involved (comparable to the manual work when A and B were integrated one after another).

And yes, picking the right strategy is a manual decision, and it will involve some time and manual work - this is not a fully "automated way", but it does also not need "complex manual intervention" every time it happens. In reality, the range of an issue fix or rollback goes from "trivial" to "complex". When aiming for continous delivery, however, one should strive to have the trivial rollbacks happen much more frequently than the complex ones.

The best approach for this is probably to pick features for parallel development in a way it makes it unlikely they need to touch the same part of the code base. Furthermore, there should be as many automated tests as possible. So the devs can run these tests during feature development before their code reaches "master". That should prevent frequent show-stoppers during QA.

Now, let us apply these recommendations to your scenario:

As long as feature A is not ready, feature B cannot be tested because it also contains the rejected code from feature A.

But why not? When A and B were eligible for parallel development, they should be mostly independent from each other and so eligible for independent testing, which means B can in most cases be tested even if the issues with A are not fixed. Lets say QA reports a serious bug from A which was missed by the automated tests. The fact the automated tests did not fail should mean at least the application should not be completely unusable, so they can start with the tests of B. They know master cannot be delivered to production as long as the issue with A is not fixed, but during the test of B, there should be enough time to fix or rollback A in the one of the 3 ways I mentioned above.

Of course, one has to make a decision what the quickest way is to remove the issue with A and make the code "production-ready" again. So when the test of B is done, a fix for the issue with A should be already available (or at least soon), so QA can approve that the issue is gone now (either with the functionality from A now or not).

Looking at the timeline, after each test cycle your "master" can switch between two main states: either it has "known defects", or there are "no known defects any more". Whenever it reaches the state "no known defects", you can deploy to production. To make continous deployment work, one has to plan the feature slices in a way the state "no known defects any more" is reached as frequent as possible. The key here is to

• make the individual feature slices like "A" and "B" from above as small as possible

• pick feature slices for parallel development as independent as possible

• plan for reverting or disabling a feature slice in a very quick and smooth way in case it contains a bug which prevents the delivery to production

In a bigger team (maybe a dozen or more developers), when you notice you reach the state "no known defects" too seldom because of the constant checkins to master, you can also consider to use an additional "pre-production" (or "staging") branch where only features are integrated which were approved by QA on the master branch. However, this does not come for free, you need another person doing the integrations on "pre-production", another QA step on "pre-production" to check the integrations itself did not introduce a bug, and additional administrative overhead for managing the pre-production environment.

Answer 2

Don't think of this as a source control problem, view it as an ops and deployment problem. You shouldn't need to commit code to fix issues caused by a bad deploy. How can you change your deployment pipeline so you can just roll back to a previous version to fix the problem?

You can look at how google and facebook do deployments. Facebook, for example, ships new features to production once a week and is generally pretty stable.

If you look at how they achieve this the key elements are:

• Code reviews and unit tests to do quality control prior to shipping
• Launching internally before launching externally to catch errors sooner
• Staged rollouts so that new versions are not seen by all servers/users at once
• Feature flags to turn new features or new versions of existing features on and off in production, without changing the deployed code. Facebook links feature flags to profiles, so that the profile of a user determines which feature set they see.
• Soft launch, leveraging the feature flag system to show new features only to a subset of users. This enables patterns like shipping features which are not yet localized to bake them with actual use.
• Dark launches, shadow production and double write/read techniques to have the old and new versions of code and database schema's running side by side in production. Users see the old versions output until the old and new agree, and then they are switched over to the new version.

Answer 3

If manual QA is part of your process, it should actually not be part of the automated release pipeline. You can still do automated builds and deployments to a QA environment, but it should be in a separate pipeline. Then, there's nothing to rollback if QA rejects something. You only send stuff to your release pipeline if it's "done", and part of that definition of "done" should be passing QA.

This also brings up a broader point that if something is being rejected by QA, there's a fundamental flaw in the process. Automated builds, testing, integration and deployment should catch all the major problems. QA should really just be doing exploratory testing, and anything found from that can just be added as backlog items for a future product iteration. Showstopping bugs should have been caught by dozens of different steps before manual QA.

Back to your pipeline, having a QA pipeline and a release pipeline means having more than one branch of your code. Though, with Git that "branch" can be as light-weight as a pull request. Your QA pipeline can then integrate the pull requests and kick off build and deploy for each one or any combination thereof (rollup). Only when the pull request passes manual QA would you actually merge it in, which would then cause it to go through the release pipeline.

If things still do go sideways, your best bet is to roll-forward. In other words, if you want QA as part of your release pipeline, then you need to commit to a strategy of not rolling back. If something is "rejected", it must be fixed and merged back in to push a new release through the pipeline. It's a bit more risky, but if you've got a good setup, you shouldn't be needing to do it often, and the fixes should be easy ones.