Choosing the right branching strategy for releases

Question

Starting with a new dev team on a new project and we have to define our Branching strategy for our source repository (e.g. Microsoft Team Foundation Server 2010). We've run into a sticky discussion over whether or not to...

A. Have one Release branch from which we do production builds and then Label when something is actually released

OR

B. Have a new Release branch for each new Release into production (Ex. Version 1, 2, 3, etc...)

Option A seems pretty straight-forward but we're not sure if this will lead to problems in the long run. Option B seems like it just creates lots of one-time long-lived branches that just pile up over time.

Does anyone have any experience either way that could help us decide? Specifically, I'm looking to hear where the pain-points are for any one choice. Feel free to provide specific experience relative to TFS and/or Release-Management implications.

Answer 1

Option A. Just using mainline and tagging for release

Pros:

• You avoid merge hell.
• Keeping to the mainline encourages some best practices like proper release planning, not introducing a lot of WIP, using branching by abstraction to deal with out-of-band long term work, and using the open closed system and configurable features for dealing with managing works in progress that may; or may not; need to be disabled now or in the future in order to release or to avoid a full rollback.

Cons:

• Dealing with works in progress becomes an issue and adds to potential surface attack area when it comes time to release. However, if your developers are disciplined then new features should be configurable and modular and therefore easily disabled/enabled, or there is no WIP and at each release point all work is either completed or has not yet been started (i.e. Scrum).
• Large scale/out-of-band changes require more thinking ahead of time to implement (e.g. branching by abstraction).

Personally I prefer this approach. Code coverage and unit tests should identify code that isn't ready to go out the door and people should not be working on code that will not be released during the current iteration. Branching by abstraction or other mechanisms can be used to deal with long term changes and works in progress.

When you don’t do this you start finding yourself dealing with merge issues, stale code, features that never get released, etc.

Option B. Branch by release

Pros:

• You can begin working on the next iteration while the current iteration finishes its round of acceptance testing.
• Other stuff im sure.

Cons:

• Tons of branches.
• Still need to tag branches at release points.
• Still need to deal with WIP and merge WIP from previous release branch into next release branch if it's not going to make it and still need to disable or yank it of release branch and re-run acceptance tests
• Hot fixes need to be applied to more branches (release branch + hotfix + new tag + merge hotfix into vnext branch and possibly vnextnext depending on where the hotfix falls.)

I'm not a huge fan of this solution ^_^.

Generally I would recommend just trying to stick to the mainline. If your developers are having trouble with not writing WIP that can be easily yanked when it fails the cut or that is checked in early for the next release then you can start talking about tagging the code at the point where it should be code complete and branching from there if necessary to address overlooked defects and bugs that your developers unit tests failed to catch.

Ideally though I think you want that to be the exception process, not the rule.

Option C. Crazy Bonus Option

If you want to get fancy you can also consider a per-user-story/per-feature branching model. (A terrible idea in TFS or any non DVCS while at the same time incredibly trivial to implement if using a DVCS like git or mercurial).

In the past I implemented the below for a previous employers maintenance team which worked with a legacy code base that could not easily be ported over to mercurial from svn. A lot of unnecessary work was involved to meet a business requirement of a always releasable mainline rather than just coordinating releases better but . . .

1. Features were developed by devs in their teams dev branch.
2. When a feature is ready to be peer reviewed the devs bundle it together into a single merge from the Dev branch into the CR branch and include the feature id/user story in the title. *Enforced by pre-commit hook*
3. After it passes CR an admin tool is used to Promote the feature into the QA branch. (I wrote a little terminal application that listed the user stories present in the various acceptance stages and allowed the operator to promote or demote it inbetween those acceptance stages)
4. QA runs automation and manual usability tests. If the feature is good its pushed into the release branch (mainline). If the feature is rejected its demoted/reverted out of the QA branch until devs can address the issues brought up during test and add submit a patch to the CR branch.
5. If code was reverted from the QA branch and a fix is applied the terminal tool will re-apply the necessary changes to bring the feature back onto the QA branch from the CR branch so that QA may re-review the code and either promote it or demote it again.
6. At any point in time the release branch should be in a stable releasable state.
7. After a release new Dev, QA, and CR are spun from mainline.

Answer 2

We have separate branches for each release we put out (appr. 4 a year). It is very convenient when you need to pull a specific release.

If you need to maintain a couple of older releases, I don't think that labelling would do. With specific release branches, you can apply hot-fixes to each branch separately (or to a selection of them) without worrying about any of the other releases.

It also makes comparing releases that much easier when you are hunting for when a bug or a feature was introduced.

Don't worry about the number of branches or about the time they go without changes. Your versioning system is to give you control and to provide a history of your project's development. History has a tendency not to change... And don't worry about your cvs not being able to cope. We use Perforce, 9000+ files in a development branch, up to 50 development branches for the release(s) we are working on and as already said, a single branch per release that we publish. Perforce is not even breathing any harder.

In short: make your life as a developer/maintainer/bug-fixer/problem-hunter easier and don't worry about the number of branches or the number of files. Any self-respecting cvs will cope.

Edit:

We don't suffer any confusion at all with regard to the number of branches we have. Our naming scheme for the release branches and our 1 issue 1 branch policy for the development (or work) branches may have something to do with that.

Release branches are named for the release they hold, ie: R2011SP1 for Release 2011 Service Pack 1. Our work branches have less intelligent names: sub01, sub02, sub03 etc. The "sub" comes from the fact that all work branches are sub branch of the acceptance branch. The acceptance branch being the one where all issues are collected that are ready to be released.

Our 1 issue 1 work branch policy, combined with the fact that our issue tracking system has been customized with a "branch" field ensures that we always know what issue is developed in which branch. When an issue is integrated into the acceptance branch this field is updated. This means we always know which issues are ready for release (once the acceptance testing is done). Similarly we update this field when a release branch is created and this way we can always track down in which release an issue released.

Answer 3

It's all about context: how often do you release and what is in a release.

Here's a bit of a case study that I had with my old work, using the B method (we called it branch by purpose).

To put the story in context,

• A release consisted in new features in our software: new gaming modes, new functionalities, new configuration options.
• The release cycle was fairly long: our clients were universities that would stick with one feature set for usually a year.

Main development was made into the trunk until we reached a feature-complete state for a certain release. At that point, we would create a branch, say projectname-january2012 and do our quality testing and bug fixes in that very branch. Once we were ready for a public release, we would tag the code in that branch, and release.

However, development on the release did not end at that tag. Inevitably, we had clients that found bugs or small issues with the release. So in that case, all we need to do is to go back to that branch, patch the code and create a new tagged version of the january2012 branch to be released, and merge the fixes back to the trunk.

In our case, this approach was favorable because some users preferred to stay with older releases with a limited set of features, or simply because the cost of deploying on their infrastructure a whole new version rather than a hotfix caused some issues.

So the questions you have to ask yourself are:

• How often do I release?
• Are my releases going to be 100% backward compatible?
• Will my clients be ok with completely upgrading to fix bugs?

If you release often, then maybe it's not worth it to have branches for each of them. However, if your release cycle is fairly long like my old use case, and that deployment, backwards compatibility and clients clinging to old releases might be risks, option B will certainly save you a lot of pain, will make things a lot easier to support your clients at the minimal cost of dealing with branch clutter.

Answer 4

I prefer option A. Develop on the trunk and branch releases when they are stable. This significantly limits the work in integrating hot fixes applied to production release.

I have been contracted to help a team that attempted option B get back on track.

A few things to consider.

• Migrate hotfixes forward through all active code branches. This can be done by merging, patching, and/or redevelopment. These should be fully managed to ensure a fix gets applied to all appropriate releases, then to trunk.
• Consider feature branches to enable developing features in isolation from the main code stream. These are advised for experimental changes. Feel free to abandon feature branches if the feature doesn't work out.
• Tag and track your merge points.
• Branch your releases when required. I find this is normally when the release is ready for release candidate builds. In some cases introducing incompatible changes to trunk may force and early branch. Consider a feature branch.

Answer 5

I've worked for some number of years on a system that uses something somewhat between the two schemes you describe. The key is that there is a multi-level numbering scheme in use. The outer level is basically the API version, and that's managed on branches (with appropriate cross-merges when something needs to be fixed on multiple branches) and the inner level is the exact releases done, which is managed with tags.

In particular, if we know what exact version a customer has, we know exactly what source the code was built from and can make an exact duplicate so that we can see exactly what is going on. This is very important for support! Yet the outer level of branches, the API versions that we currently release, they do evolve over time (with the main trunk of development getting the majority of new features). Also, when we do a new major release of the API, we do a new branch for supporting that from (so the trunk can always be hard-core development oriented) and we consider whether we should end-of-life the current oldest support branch.

Thus I recommend something that's really a mixture of both A and B; both have good aspects, but neither is complete in itself. Use the best of both worlds.

Answer 6

I've used TFS to effectively implement option (B) in the past.

Branching/merging is a great tool when it's done in small pieces. The difficulty doesn't lie in making a branch (that's stupid easy), nor pushing a week's worth of work back up the tree (that's usually easy too)... it's in getting the CI system behind your source control to automatically work for you.

Because branching is moot if the system isn't automatically building and running tests for your branch.

We had customized TFS' default build workflow to recognize changesets' relative paths, and established a convention by which the customization could recognize a new branch (as opposed to simply a new subfolder under some development root). It was smooth, easy to branch, easy to kill a branch, and we got continuous feedback from our system for compiles and tests.

I see a lot of people declaring how impossible these strategies are under TFS, and I believe it's due to a lack of familiarity with the possibilities of a XAML-based build engine. TFS isn't just source control, it's a whole solution, and should be used as such.