Looking for some alternative branching strategies to meet my team's needs

Question

I've recently joined a new company and they've been experiencing some issues related to their branching strategy which causes untested code to be released inadvertently.

We have three branches of interest:

1. main: this is the branch we release from
2. develop: this is the branch that contains code that is yet to be tested and merged into main to be released. We deploy a development environment from this branch and this is also where QA is done
3. feature: this isn't an actual branch - these are branches that individual developers branch from develop and work on a specific feature to merge into develop

Here is one issue we're facing. Let's say we have developer A and B working on feature A and B, respectively.

1. Developer B finishes first and merges feature B into develop
2. QA begins testing feature B
3. Developer A finishes feature A and merges it into develop
4. QA finishes testing feature B but they do not have time to start testing feature A because our customer's deadline to release has been reached
5. Devs prepare the release by merging develop into main and releasing the stuff on main

The issue here is that main now has feature A which is untested!

Usually the way this is addressed is that a "hotfix" is made where feature A is reverted on main. However, I think in this case prevention is better than the cure. What kind of branching strategy would you recommend so that we don't end up in this position where work has to be reverted on main so it does not end up in the next release when it shouldn't be?

Answer 1

Others have described various approaches of what you "should" do. But if you get insurmountable pushback on those suggestions from your team or management, here's a de minimis change which should fix your main stated issue:

Add a new qa branch.

Here's how the workflow plays out:

1. Developers make new features in branches featureA, featureB, etc. (This is unchanged.)
2. When the development group thinks the features are ready, they merge to develop. (unchanged)
3. When QA is ready, they pull down develop (unchanged), and merge it to the qa branch. (new)
   • This should be a fast-forward merge. (There's no separate merge commit - both develop and qa point to the same commit ID immediately after.)
4. Developers can continue to merge to develop while QA is running. (unchanged)
5. If QA passes, the QA team (or the dev team on their behalf) merges to main (unchanged), but they merge the qa branch, not the develop branch. (new)
   • This should be a fast-forward merge.
   • The state of main after the merge (including commit ID) should be identical to the state of qa which passed QA.
   • None of the state of develop which isn't already in qa gets merged to main.
6. If QA fails, it gets kicked back to the dev team to fix. (unchanged)
   • How you handle this now depends on your release timeframe and appetite for risk. (probably unchanged) You can either fix it in develop, or hotpatch the qa branch.
7. If you fix the QA failure in develop, go to step 3 and repeat. (probably unchanged)
8. If you hotpatch the qa branch, you now have diverging histories in qa and develop -- fix that by merging qa back into develop. (new)
   • If you're not too fussy, you can just do a regular merge.
   • If you're a "linear history" shop, have your git experts rewrite the history of develop to be linear & stacked on top of the qa branch state.
   • The key thing is to make sure that there are no commits in the history of qa (or main) which aren't also in the history of develop -- the merge in step 3 (and step 5) is always a fast-forward one.

---

Is this the ideal way of dealing with branching? No, I'm not claiming it is. I'm simply suggesting it as it's likely one of the smallest changes to your current workflow which fixes your merge-to-main-and-revert-untested problem. If you can't get the team to buy in to other, "proper" fixes, you may be able to get them to buy in to this one.

Answer 2

Graydon Hoare popularized the Not Rocket Science Rule of Software Engineering:

The Not Rocket Science Rule Of Software Engineering:

automatically maintain a repository of code that always passes all the tests

It means: design processes so that untested code is never merged. Instead, merging is a consequence of successful QA. (More precisely, a merge is done, that state is QA'd, and then the respective branches are forwarded to the merge commit. If success is likely, multiple changes can be tentatively merged and QA'd together.)

Your process violates this rule because QA only looks at code that is already merged.

There are a couple of locations in your process where you could try to introduce a proper qA gate.

• Full QA only on release candidates.

  In your current process, QA tries to cover all commits in the develop branch, and the tip of that branch is always used to cut releases, regardless of the QA status of the tip.

  Instead, QA could branch off a release candidate from the develop branch, do QA on that while work on other features continues, and work with developers to fix the release candidate. At the end of this process, the release candidate is merged into the main branch as a release. The fixes in the release candidate also get merged back to the develop branch.

  This happens to be exactly the Nvie Git Flow. It is well-suited for infrequent big releases that involve lots of manual QA, but it would require a lot of unnecessary ceremony if you're aiming for continuous deployment.

  Also note that such manual processes violate the Not Rocket Science Rule.

• Better QA when merging features.

  Make sure more testing is done when features are ready to be merged, to ensure that the tip of the develop branch is always in a read-to-release state. Ideally this is managed via an automated process following the Not Rocket Science Rule, but in principle it can also be done manually by a person who confirms that QA has passed before promoting the merge commit to the develop branch. For this to work properly, only one merge/test can be in progress at any given time.

  This gets a lot easier if as much QA as possible is automated, and if the entire QA process only takes a couple of minutes.

• Dual Approaches. To some degree, you can combine both approaches, where you have high confidence that the develop branch is in a ready to release state, but still gate releases on a manual QA processes. This can be appropriate were a full QA processes on every merge would be too expensive, or too slow.

• Feature toggles. Just because some functionality has been merged does not imply that it must also be activated in a release. Build-time or run-time feature toggles can be part of a solution here. For example, developers might guard new functionality behind a toggle and disable it by default. Once QA has passed, the toggle can be enabled by default and eventually removed.

  There are of course many drawbacks to this, such as the difficulty of ensuring that all new behaviour is properly protected by the toggle, the interactions of different features and the combinatorial explosion of such interactions, and accumulating cruft of old feature toggles that are no longer used.

Answer 3

You may want to use the Branch For Release strategy. That is create a new branch for every release, and then QA's will do testing on that branch. Once the release branch is created normally the only changes on it would be bug fixes based on QA feedback.

Based on your current system, the release branches would be created from develop, and then either merged into main or directly released without ever being merged. In the latter case you could get rid of main and then perhaps rename develop to trunk, main, or master.

Any features merged into develop after the release branch is made will simply miss that release and end up in the next release - unless you decide you want to do an additional merge or cherry pick into the release branch sometimes.

Answer 4

As I see it, there are a couple of issues here. This goes beyond your branching strategy.

4. QA finishes testing feature B but they do not have time to start testing feature A because our customer's deadline to release has been reached

A couple of things have gone wrong here. Your QA department seems to have over-committed regarding the amount of work they could get done during this cycle. Whoever is managing this project hasn't been diligent about evaluating the work time remaining against the amount of work QA has left to do. If they had, they would have seen that it was becoming unlikely for feature A to be tested on time and would have bumped that feature to the next milestone. Your process also appears to be missing the step where changes on develop are frozen some period of time before the deadline (enough time for QA to finish their qual cycle on everything that's in there).

5. Devs prepare the release by merging develop into main and releasing the stuff on main

Merging develop into main is fundamentally incorrect. You should merge the most recent revision of develop that has passed all QA tests into main.

Fundamentally, I think this is more of an overall process problem than a branching strategy problem. It feels like you have multiple actors doing their own thing, with no real planning or coordination between them. No branching strategy is going to work if that's the case. Here are my recommendations for your overall process.

• Plan, in detail, what work will be done in each iteration. This not only includes the time required to design and implement the feature, but also to test it. Multiply all estimates by 1.5-2x before using them to make a schedule.
• Every couple of days, compare the list of tasks remaining against the amount of time you have left in this iteration. If you get behind schedule, start deferring work items to the next iteration. Do this early and often, you can always pull things back in if you end up with extra time.
• Don't merge a feature branch simply because the developer is done. At a minimum, it shouldn't be merged until it passes basic sanity tests done by the developer, has passed a code review, and has a commitment from QA that it can be tested in this iteration. Until then, it stays in a feature branch. It might stay there until the next iteration, and that's OK.
  • A system based on pull requests can make this easier to manage and enforce. You can set up policies so that a pull request cannot be completed until all relevant parties (QA, the code reviewer, the manager in charge of scheduling, etc) sign off on it.
• Stop working all the way up to the customer's deadline. That's the deadline for delivery. You need to have your own internal deadlines that are much earlier than that. Development needs a deadline after which the code is frozen for that release (except for critical bug fixes). QA needs a later deadline for when the frozen build should be completely finished with testing. Your release team needs an even later deadline for when that fully-tested build needs to be packaged and ready to deliver. Measure how long each of these steps take, then work your way backwards from the delivery deadline and build a schedule of internal deadlines to work to. Ending all QA and development work at the same time practically guarantees that you'll have untested code escape.
• Always keep track of which build QA is currently testing, which have been fully tested, and which haven't been tested yet. When it comes time to do a release, you'll use the latest build from that "fully tested" category. From a mechanical standpoint, you can keep a tag pointing to the latest build that has been fully tested. Releasing would then involve merging everything up to that tag instead of the entire develop branch.

Answer 5

Devs prepare the release by merging develop into main and releasing the stuff on main

The issue here is that main now has feature A which is untested!

Old joke:

• Patient: "Doctor, doctor, it hurts when I do <a thing>"
• Doctor: "Well, don't do <that thing> then"

Seems to me your whole problem is solved if you cut the release from the point on develop which contains the last tested feature, not "whatever happens to be at the head of develop when we do the release".

Answer 6

Builds get promoted not branches. Builds are based off of commits. Commits can be direct or via merging. In your situation: Main is your current production code. Develop is next release. If you only want feature A, then merge only that commit into main. Not the feature B commit.

At certain points you may want to introduce a soft merge freeze to develop to avoid something that was developed that you don't necessarily want in the next release to production. Developers continue to write code for the next features they just don't merge things on develop since there is a soft merge freeze. This allows to add in any last minute bug fixes/changes prior to release. Usually this soft freeze is only a few days in duration. Once the code is released, developers can start merging to develop again because main will represent production.

You could also create another branch after making the cutoff if you expect a lot of fixes/changes from the time you created the production candidate release to actual production release. But you will have to merge that branch to production and back to develop for any changes done since cutoff.

If you need a hot fix after release, then you make changes to Main and then merge backfill to develop.

So, some sort of management is required to keep things in order. If you do trunk based development, every commit is a commit that can go to production. If you make a mistake, it's just another commit and push to prod. But most organizations have some sort of develop/test/quality assurance cycle. Someone should be directing what is/is not is going in and doing the appropriate merging at the specified intervals.

Some of this depends on timing. Let's say I am releasing every 3 weeks. On day 18, we have a soft freeze. So, only 3 days of not merging. Creating an extra branch is probably not worth it. Now consider a company that releases once a quarter. The freeze may be a month before actual production. In that case, if makes sense to cut a production candidate release branch since there will be 30 days to production and we don't want a code freeze for that long. Just remember the production candidate branch needs to be merged to production and to develop. So, some of this depending on your individual circumstance.

A final distinction is developers create branches off of one of these "named" branches. A hot fix branch would be created off of main. Next features off of develop. Any last minute changes fixes off of the production candidate branch.

Answer 7

This is one of the reasons why I really dislike the whole long-lived feature-branch idea. The idea that you develop huge features in isolation in your feature branch and merge into a main branch once it is done, looks nice on paper. But in practice it provides more issues than it solves. Firstly long lived feature branches are problematic as the merge hell of getting the change back on main is costly and error prone. Secondly you are hiding modifications from other developers that might have to adjust their change.

First of, I would change from merging developer branch to main branch on release to instead cherry-pick the changes ready for release from developer branch to main. Basically you only want to pick the changes approved for release. That is a short term fix for your issue.

But more importantly I would change the development mantra. I would break up the big changes, instead commit often and push to the development branch. You might ask whether that would not lead to half completed changes pushed to development. And that is basically correct, but that is why you branch by abstraction instead. Use feature toggles, or push changes small enough to not in them self break other things. By committing often you lessen the merge hell of getting a huge feature to main.

Also tests are more efficient and more thorough when the scope to test is smaller. So during your development QA can test parts of your feature as you develop it. So once the feature is done, most of the functionality has already been tested, but more importantly other developers have had the chance to test their modifications against your changes, and vice versa.

Finally remember to pull and rebase from the dev branch often.

Answer 8

I have an idea for a branching strategy that could solve this issue that I would like to get some criticism on.

Again, you still have main, develop, and feature branches, except this time, this is the developer workflow:

1. Create your branch from main
2. Ensure your branch passes all unit and sanity tests
3. Merge your branch into develop
4. Wait for QA to test develop
5. Merge your branch into main
6. Your feature will be in the next release

I believe this solves the issue of untested code ending up in main. But I'm wondering if it introduces some other issues.

Answer 9

Don't merge latest develop into main - merge the exact commit that was tested.

Always keep track of which commit is being tested by QA. You can usually export the short hash in a build output or append it to the version number. Or someone can manually record it.

Then when you're ready to release, make a dedicated prerelease/<version> branch from that tested commit hash instead of latest develop.

Answer 10

I think you know what the problem is. You have one too many branches.

Get rid of your develop branch. For this is the way to developer hell

Why have a develop branch when you need to do QA on each new feature?. make sure the branch is up to date with master, hand the feature branch to QA, when they are done complete the pull request to master. Don't merge code until it has been tested/qualified. As long as you do it this way (assuming QA is a bottleneck) no merged code will be untested/unqualified.

This way you will only need to merge once (getting the feature onto master, right before the feature is ready for qualification). Instead of merge to develop branch, have QA run on that feature amidst all of the other features that are not yet qualified. fix the problems detected by QA, merge again, do QA, fix the problems generated by other peoples code that hasn't been qualified but the merged in between QAs..., merge, QA again, find more problems fix them, someone merged more un qualified code... This sounds like a nightmare.