What are the risks of having a release branch?

Question

Currently we manage our source code in a Microsoft Team Foundation Server project. We have the following branches:

• Development
  • Feature 1
  • Feature 2
  • Etc.
• Test

The process is that development is either done in the Development or a separate feature branch. As features and fixes are completed they are merged to the Test branch that is then deployed to a test environment. When we have a successful acceptance test of the Test branch we create a release from the test build and deploy it to production.

The reason for this approach is that we did not want to do a merge from Test to a Release branch before building and deploying to production since we would deploy different code than what was tested.

The downside of this approach is that often there are untested changes waiting in the test branch which delays delivery of hot fixes for critical bugs in production.

I see that it is common in the industry to merge to a Release branch and then deploy that. Is there a method that allows for easy hot fixes and not introduce the risk of untested differences when having a separate release branch?

Answer 1

...often there are untested changes waiting in the test branch which delays delivery of hot fixes for critical bugs in production.

Is there a method that allows for easy hot fixes and not introduce the risk of untested differences when having a separate release branch?

Yes, there is. The problem you've encountered is being caused by a bug in the workflow:

feature1 \
feature2 --> test --> release
hotfix1  /

As you've seen, if the features take time to test thoroughly, they act as a barrier for getting the hot fix through the pipeline. The takeaway there is that if you need to get a release out quickly, there shouldn't be anything in the release pipeline that isn't very close to being ready.

You can prevent that by moving the barrier ahead of the release part of the pipeline so that any delays in testing a change stay with the change:

feature1 --> feature1-test \
feature2 --> feature2-test --> release-test --> release
hotfix1  --> hotfix1-test  /

When a change is finished, it goes into a separate branch where it is tested. As long as testing is underway, all work on that change happens between the two branches. After testing has approved it, the tested code is merged into a pre-release testing branch where the full test battery is run once more before release. (In theory, this should be a formality and a non-event; I'll explain why later.) This means you can take a long as you want testing feature1 without delaying feature2 or the hotfix.

One important thing that's frequently overlooked is feedback. Whenever a release happens, that code needs to be merged into all branches where development and testing are being done. If hotfix1 breaks feature1, you want to find that out in feature1-test so it can be fixed there and won't stand in the way of a release containing hotfix2. If the feature integrates the released changes early, there should be fewer surprises in release-test. This arrangement allows you to treat hotfixes and features the same way from a development perspective and leaves the decision whether to subject a hotfix to the full battery of testing or fast-track it a matter of QA policy. The workflow is the same no matter what, and there isn't a quick branch/merge from/to the release branch.

NB: What's described here is a bit of an oversimplification because it doesn't take into account something in feature1 breaking something in feature2. This arrangement can be used in more-complex structures where release-test becomes integration-test and that feeds release-test.

Answer 2

In an environment like yours, with your branching strategy, production hotfixes should occur on hotfix branches off the release branch. Once tested and approved, those changes should be merged back into your upstream development and test branches. This is a highly conservative branching strategy. And the extra merging is one of the trade offs you make to have that level of safety in your releases.

Answer 3

The root cause of your problems is the use of feature branches (some people, myself included, consider them "evil"). Merging them together to obtain your single release branch is your bottleneck: branch merges are discontinuity points in the quality levels which can not be avoided, require unknown amounts of work/resources, must be serialized, can not be properly planned, etc. They cause that untested/in-progress feature backlog which becomes obstacles between your hot fixes and your release branch.

People often forget that a change properly working in one branch is not guaranteed to be working just as good (or at all!) in another branch. An entire feature can be working just fine in its own development branch, only to be completely ruined when merged with all the other features that need to be released. And you can't even pinpoint which individual change is causing the problem: you have to take an all-or-nothing approach: either you cancel the merge (i.e. not integrating any of the branch changes) or you accept the merge (with all its changes), taking the hit in the quality level which you then need to address, always without any potential suspect in mind.

In a Continuous Integration approach there are no feature branches to be merged - no backlog, no such obstacles. Any and all changes are kept small and can go into the single branch at any time, regardless of being a hotfix or part of a new feature. Faulty ones are immediately identified and addressed, automation can help immensely (yes, in CI there is always a very short but never empty suspect list). There are no quality level discontinuity points, quality is constantly being measured/monitored and can even be enforced automatically, via tools.

Even if you can't really use your single main/master branch as the actual release vehicle and you do need release branches (perfectly normal when continuous deployment is not an option - for example in embedded software or OS development cases) you can still use continuous integration at the release branch level. That means you never rebase/reparent a release branch, or merge into it some other feature/integration branch to bring in a bunch of changes which appear to be working fine (in that other branch's context).

Each change, hotfix or even part of a "late feature" development, developed specifically for a release branch or ported/double committed from some other branch, goes in through the same checks applied in the particular context of the receiving release branch. This ultimately guarantees the minimum quality level for that release. Which is why I believe CI/CD is the fastest way of reliably releasing software.

Answer 4

In general you have 2 version control approaches that you can take to handle this issue:

1. GitFlow

Here you have long living feature branches that get merged in the development branch when they are done. The development branch is merged in the master branch when all the features for the next release are done. When a hotfix needs to be applied a new branch is created from the master branch, the hotfix is commited on that new branch and that branch is merged into master and develop (you then deploy the master branch to your test environment, test that version and then deploy it to higher environments).

A feature branch that took a while to develop can have many merge conflicts. This can be mitigated by often rebasing the feature branch on the develop branch (or by merging the develop branch in the feature branch).

2. TrunkBased development

In trunkbased development there are no long living branches, there only is a master branch. So each commit is on the master branch, and gets deployed to all environments imediately. Feature flags are what prevents a change from being visible in production when it has not been tested in earlier environments. In this way you don't have the hard work of solving merge conflicts. You however get the overhead of managing feature flags in its place.