As I am looking more and more into Continuous Delivery for our organisation, I struggle to understand how we can effectively incorporate manual QA testing before a release is pushed out into production.
I have found multiple approaches, such as the branch-per-feature one explained by Adam Dymitruk or git-flow and the likes, which all seem to include a lot of manual work to get things setup each sprint or perform rollback operations.
I'll try to explain my issue by given an example.
Most CD documents outline the "happy" path flow as follows:
develop -> CI build -> deploy to QA -> test on QA -> promote to Production
This is easy enough to understand and works fine if there are no issues. But let's assume the following workflow which requires manual testing before a deploy to production is authorized: developer 1 works on feature A, developer 2 works on feature B. I use the name features, but I don't necessarily mean this in the broad range of a feature which could be contained by using feature toggles. Just some work.
developer 1 finishes his work, merges everything into master - or the equivalent release branch -, triggers the build which is deployed into QA. QA starts testing. During QA tests, developer 2 finishes his work, merges into master, triggers the build. QA then rejects feature A. Thus, the release from developer 1 cannot continue. At this point the entire process halts. As long as feature A is not ready, feature B cannot be tested because it also contains the rejected code from feature A.
And this is the part which I don't really understand how to fix in an easy, automated way. How can we effectively rollback feature A without complex manual interventions? How do people do this? Surely this must be an issue everyone faces, but none of the books or schematics I've read so far seem to really address this issue.
Taking it one step further. GitHub flow seems to suggest to deploy the feature branch into production, and rollback if a production issue occurs. That might work in a single developer single tester scenario, but what happens if multiple features are signed of by multiple testers at the same time?
developer 1 starts his feature, branches of master, starts working, pushes into master, QA 1 tests. At the same time, developer 2 went through the same process and QA 2 tests his feature build. Given the schematic, feature A does not contain feature B and feature B does not contain feature A. QA 1 signs off, feature A goes into production. QA 2 signs off, feature B goes into production. But feature A is lost. It's in master, yes, after the deploy it had been merged, but it won't get back into production until feature C gets deployed.
How would GitHub flow effectively work in larger teams? Has anyone some practical examples on this?