How can we only include ready-to-be-released features in our production releases every other week?

Question

I'm a software developer on a fairly large agile team (we have eight developers actively making changes to a single code repository). Every two weeks, we push a new version of our software to production. Here's our current workflow:

• When starting a new task, developers create a "feature branch" off of the main development branch (we use git) and work off this new branch
• Once a developer has finished work on their task, they merge their feature branch back into the development branch
• The developer merges the development branch into the QA branch.
• A build is triggered off the QA branch. The output of this build is deployed into our QA environment to allow the testers to begin their testing.

It's quite common for our testers to find issues with these new features that have been merged into the QA branch. This means that at any given time, the QA environment likely contains several new features - some tested and bug-free, and some broken. This makes releasing difficult because it's rare that the QA build is in a production-ready state.

To mitigate this, we've been trying to initiate a "QA freeze" which means developers don't merge our development branch into the QA branch a couple days before the release. Bug fixes to the QA environment are made directly on the QA branch and merged down to the development branch. Theoretically, this keeps new, broken features out of QA while still allowing us to fix issues already in QA.

While this "QA freeze" concept has been partially successful, it's hard to coordinate and people are often confused about whether they are allowed to merge to QA. It's also been hard to set a "QA freeze" deadline - everyone likes the idea of some breathing room in between the freeze and the release, but in practice, they'd rather have their feature in the next release than respect the deadline.

Is there a better way to ensure that we have a clean build for our releases every other week?

Answer 1

There are a few problems floating around in this that are causing issues that you are experiencing.

The first is the long running QA branch. Having a long running branch that is parallel to the development mainline may as a source of confusion because there are different efforts that need to be replicated in both the QA branch and the mainline. This means that either you are checking in fixes to the QA branch that need to be merged to the mainline (not a bad thing), or you are checking in to the mainline that gets merged into the QA branch (a source of possible bugs).

The other problem with the long running parallel branch is that it is possible for files to become perpetually out of sync. A code fix that never gets merged back, or a configuration needed for production builds that is never tested and part of the development mainline.

Next, you've got roles that are getting impinged on. This means that the packaging role (more on this later) is not getting sufficiently isolated.

In the git-flow model, the release branch is branched from development (not development merged to QA) and all fixes are checked into the release branch and then merged back to the development branch.

Some of the philosophy of branching can be found in Advanced SCM Branching Strategies (I consider to be an excellent read). This focuses on the roles that each branch may take on. The release branch takes on the packing role.

The packaging role is often confused with the accumulation or, more commonly, mainline roles. Once the intended development and maintenance have been performed and any accumulation has been done, it is time to prepare the code for release. Such an effort may not be trivial, requiring a team of release engineers and additional fixes beyond those already performed. The policy on a packaging branch is significantly different from that on a maintenance branch, as the packaging role suggests, only the changes necessary to make the product releasable should be addressed.

• Branch from the development point to the release branch. The release branch that QA builds from gets one branch and no merges from development.
  • If you want to go down that road, with consistent naming and hooks, it is possible to prevent a merge from being done into a release branch.
• Fix everything that needs to be fixed in the release branch and merge those changes back to the mainline.
• At the end of the release effort, merge the release branch into the "releases go here" branch and tag it as such.
  • Some sites don't have a "releases go here" branch and just leave the end of the release branch dangling with a tag.

One should seriously consider applying the entirety of git-flow in place. This isn't too far from what is being done currently and puts some discipline and consistency into what each branch means and how each branch interacts with others.

Answer 2

The problem seems to be to me that you have a single QA branch.

For each release, make a separate QA branch from the primary development trunk/master. Then merge in only fixes for bugs for features on that branch - never new features. Have QA test that branch.

This way, the "freeze" is quite evident- it's in the branch name. You could use something like, I dunno, release/26/10/2015. Then it's obvious that nobody should be merging in new features after this.

It's especially helpful if you don't even fork the branch until the freeze. People can merge in to master at any time, it just won't be part of this release if it's not done in time for it to be tested.

Do not have a single long-running QA branch, that's just begging for trouble. Fork from the main development branch for each release and QA that branch.

Answer 3

You are somewhat mapped to the Development-MAIN-Production branching model seen below. The area above MAIN is said to be the development area. The area below MAIN is the production area.

Highlights of this model which I consider relevant for you:

• Your devs needs to Forward Integrate (FI) (FI = merge away from MAIN) frequently (2-3 times a week) into their DEV branches to ensure their latest changes always consider the latest overall developments.
• Your devs need to Reverse Integrate (RI) (RI = merge towards MAIN) in the TEST branch only when they've reached a feature-completion milestone which they want to expose to QA and for which they're ready to provide prompt fixes in response to QA feedback. The fixes will be performed on TEST branch and immediately FI in their DEV branch.
• Never RI from any DEV branch into MAIN
• Always RI from TEST branch into MAIN, exclusively when your QA considers the quality of TEST being OK. Keep a high quality threshold for merging into MAIN. At the very least, your Product Manager must be able to always demo a working version of your Product from the latest commit in MAIN.
• Create branches in the production area only as needed. Your build server should always tag all branches, including those from development area, and the source of any build/release should be identifiable at all times regardless of the branch it came from.
• Take releases for production only from MAIN or the production area. If later you need to provide a fix for an exact released version (i.e. you can't just give the latest version from MAIN), create a branch in the production area from the MAIN tag of the faulty release, when the fix is needed. Always fix the problem on the HotFix branch and then immediately RI into MAIN and FI into TEST.

I suspect you have problems because:

• Your devs RI into TEST code that is not feature-milestone complete
• Your devs RI into TEST without getting the green light from QA (i.e. QA is not in control of what gets injected into TEST)
• When QA reports a bug on TEST, your devs fix it on their DEV branch and then RI into TEST. This is a major bad practice because the merge will always bring in other incomplete dev crap. They should always fix it on TEST and then FI into their DEV branch. If it's not fixable on TEST, they delivered total crap in the first place and you have bigger problems.
• Your devs don't FI often enough from TEST and so they destabilize TEST whenever they deliver there. It's a fine art balancing how often to FI into DEV. Postpone it too much and it will be extremely costly & risky right before delivery, which you never want. Do it too often and you don't get any actual dev work done if you overlap too much with work delivered by other people in TEST in the mean time.

Answer 4

As I understand the question you have two problems. (a) broken features are being merged in with the good features you want to release; (b) you want to be able to release the good features while holding back the broken ones. As a constraint on possible solutions, I assume you want your final/official QA testing to happen on an integrated branch that contains all the features slated for the next release.

Regardless of your SCM branching model, I suggest you try one or both of the following:

1. Assign a QA resource to each feature team. Have them do some feature testing on builds from the feature branch, and give them authority to decide when a feature is good enough to merge. Ideally, have them work collaboratively with (the rest of) the feature team, so stuff is tested shortly after being written. (Note this does not mean they have to do all the testing themselves.)
2. Use feature toggles, either instead of feature branches or in addition to them. Done right, feature toggles allow you to turn off a broken feature without trying to un-merge it from the code, so you can test and release the other features. The kind of toggle I'm talking about is not accessible by customers; you don't want an exponentially rising number of combinations to test. You set the toggles on the QA branch to match the features you're planning to release, and if the plan changes because a feature isn't ready, you change that toggle.

Answer 5

One very simple solution which I've seen work on a team a little larger than yours is to have everyone work and deploy from a single branch.

You say the team is agile but it's not clear if you're working in sprints (ie Scrum) or a more continuous flow (ie Kanban) approach. Assuming you are doing sprints the aim for the team is to have the code releasable at the end of each sprint, for your fortnightly release. There is no confusion as to whether one feature will break another as they have all been developed together. Testers may be able to get access to features in smaller chunks as the overhead on developers to deliver to them is lower. And you don't really need a QA-Freeze, instead everyone knows when the end of the sprint is and shouldn't take on work they can't finish, or leave in a deployable (ie disabled) state.

Obviously there are pros and cons to any approach, I present this as an option not necessarily the 'best way'.

Answer 6

The reason you're getting these problems is because your code released to QA is not good enough quality (and is anyones?!), so you need to start getting a better release to QA so they do not need to receive bigfixes so often. the simplest way to do this is to introduce an intermediary branch you release to (lets call it test). This is still under the development remit, but it allows developers to push to it to continue working, while also having an integrated branch that should be good enough quality to be sent to QA.

Integration testing can take place on this branch in order to find the bugs QA are currently finding, bugs can be fixed on the original branch and then merged in again, and again until right or bugs can be fixed on this branch directly (I recommend the former). Once its passed a load of basic tests, it can then be sent to QA for the 'user sticky fingers and the they-did-what?' testing.

So this approach is designed to protect the QA branch from broken development features - whether that's because the feature wasn't coded well enough or whether there were unexpected integration issues doesn't matter. Only dev branches that pass the integration testing get to be promoted to QA.