Release strategy for multiple Git feature branches being tested simultaneously

Question

We have three environments: Dev, UAT and Prod. We use TFS to schedule releases of our master branch into Dev for our internal verification, then UAT for business verification, and of course finally to Prod once approved.

We've recently adopted a new lightweight Git branching strategy as follows:

master is always prod-ready. At any point master should be able to be deployed to production.

All new development is done in a separate feature (topic) branch as follows:

1. Create a new feature branch off master, call it FeatureA
2. Develop FeatureA until completion
3. Once FeatureA is finished, release it to Dev, and then UAT
4. Once the business signs off on FeatureA in UAT, it's considered prod-ready. Merge FeatureA into master, then deploy the new master branch to Dev then UAT. During the way, "smoke test" the branch in UAT to ensure the resulting merge into master didn't cause any unforeseen side-effects. Once smoke-tested, release to Prod.

The problem we're coming across right now is that we may have multiple features being developed in parallel, all of which could potentially need to be deployed to the test environment for verification at the same time. The approach we've taken to solving this problem is:

If FeatureA and FeatureB need to be in UAT at the same time, then:

1. Create a new branch, FeatureAandB, which will encompass both features
2. Merge FeatureA into FeatureAandB
3. Merge FeatureB into FeatureAandB
4. Release FeatureAandB to Dev, then UAT

The downside to this is that it's unlikely both FeatureA and FeatureB will be UAT verified at the same time. If FeatureA is verified and FeatureB is not, we need to release FeatureA to prod without FeatureB. What we've discussed in this scenario is to:

1. Merge FeatureA (not the joint branch, but just FeatureA) into master
2. Release master to Dev, then UAT for a quick smoke-test, and finally Prod
3. Once in prod, re-release just FeatureB to Dev then UAT so testing can continue.

The downside to this is that it directly impacts any testing for FeatureB, and potentially unwinds any work the testers have accomplished with FeatureB.

How do you manage multiple features living simultaneously in each environment and being released potentially independent of one another? We can mitigate the issue a little more if we have multiple environments, or turn-around UAT testing much quicker, but at the end of the day the same problem can exist.

I'm not opposed to hearing alternative branching strategies, either.

Answer 1

Ahh. Congratulations! You’ve destroyed one bottleneck and discovered the next one! Now it’s time to look at actually continuously integrating your code. As you’ve found out, it’s hard to continuously deliver when your code under dev isn’t being continuously integrated, but how do you make this work?

No more feature branches.

No. Seriously. No more feature branches.

It’s time to introduce feature toggles into your system. Instead of delivering new features by delivering code, you need to have complete control over when a feature is turned on for a given environment. In other words, you want to have environment specific configuration that turns features on and off. This decouples code releases from feature releases.

Answer 2

Test features separately and in order, that's really the best way to do it. Usually whichever one is done first gets tested first, but business needs could change that priority. When FeatureA is accepted, it's merged into the main branch (dev, UAT, or master in your case, as applicable), that branch is merged into FeatureB and then feature B is tested. A is tested first, then B, then C, etc, with each feature branch getting updated as other features are tested and accepted.

By testing features separately, you keep the focus on one feature and one team is responsible for fixing any problems. There's a clear responsibility for keeping the code in a good state when handling merge conflicts (in this case, the featureB team needs to merge dev in and deal with any merge conflicts, ensuring that the code from featureA is maintained). Separate features also makes it easy to assign "blame" when a problem is discovered - not the "why did you break this" kind of blame, but the "this feature is causing a problem, remove it from dev until we can sort it out" kind of blame

If you're finding that a lot of FeatureX branches are spent waiting for testing or multiple branches are being completed at the same time, you can either break them up into smaller features, or allocate more resources for testing. Both approaches should decrease the time to test an individual branch.

Similar features that will be completed at the same time may best be developed on a single branch. That's kind of what you propose with the FeatureAandB branch, but in this case, they get accepted or rejected as one. If the features are not similar / related, though, test them separately.

Finally, make sure you're making use of any automation available, so teams aren't waiting on another team for anything. Most build servers can be set to build Pull Request branches, producing a FeatureA+dev build which can go to QA for acceptance and integration testing. The developer simply opens the PR (which also serves as a code review, if desired, and also serves to signal that the feature is ready for testing), and once QA signs off on it, completes it. This then merges the branch into dev automatically. Can do the same for UAT and master, creating PRs for merging dev into UAT, and UAT into master.

You may then be able to reduce the number of branches, and stick with dev and master, plus featureX. Get rid of UAT, as that role is essentially served by dev. Actual development is done on featureX and production is in master. Deploy new code whenever master is updated, or deploy whatever is in master whenever you want to release.