6

At $DAYJOB, we have the following release/delivery pipeline:

  1. Get new features, work on new features.
  2. Once all features are completed, our own QA performs regressions, etc.
  3. QA signs off on first UAT build, ships to customer UAT department (this is really customer's own QA force). This begins our initial version (e.g. v1.0)
  4. Customer QA (UAT) finds issues, sends us back a list.
  5. Development addresses the list of issues. Sometimes they might ask for non-defects, such as improvements or small features. We triage these on a case-by-case basis. Our own QA also verifies the fixes. They sign off, we ship a new build (this gets a version bump, like v1.1)
  6. This "get list" and "work on the list" process cycles a few times, until customer signs off on the build (UAT Pass).
  7. Signed-off build becomes "production". Now developers can start working on new features for the next release.

Some notes on this process:

  • We use a bastardized version of git-flow for this process (will explain in more detail later).
  • Regarding versioning, we do not differentiate between UAT releases and production releases. We have to assume any release to UAT might be signed off and go gold at any time. For us, each build "out the door" is treated as a "production release".
  • Developers work on branches off of develop for new features. We even work on features for the next release. However, we do not merge features until they are scheduled for release. This is because we do not create 'release' branches (more on this later).

Here is how our development process goes for both bugs and features:

  1. Developer branches off develop and begins work.
  2. Developer finishes work, performs code review
  3. Code review passes, and we indicate to QA the branch is ready for testing. Our CI build system builds every branch in the remote repository, and QA can grab ad-hoc builds off of any branch. QA tests features in branches (and not off of develop) for a few reasons:
    1. Builds on develop might include multiple features or bug fixes from multiple developers over the course of some time. When regressions occur, it becomes more difficult to track down what caused the regressions. The "test-each-branch" model allows us to isolate regressions to the changes on that branch (i.e. the one feature or bugfix).
    2. All features must be tested on a branch because we do not merge them into develop until QA is happy with it. Also QA controls deployment schedule, so they know when develop is ready for new features.
    3. We're tight on QA resources. At the moment we have a 4-to-1 ratio of developers to QA personnel. They simply can't keep up with the work we do.
  4. Once QA is happy with the feature on the branch, they mark it as "verified".
  5. Once verified, it sits there unchanged until someone with appropriate authority "approves" the feature or branch. At this point, a developer rebases/merges latest into the branch, fixes conflicts, then merges it into develop.

When we finally release software (new UAT build usually), we do a merge from develop directly into master.

Symptoms of the current system:

  • Tons of branches sitting around waiting to be merged. Over 100 at the moment, and the turnaround time is sometimes months before they are merged.
  • Process is complex and tightly controlled by QA
  • Throw submodules into the mix and you've got a recipe for disaster and complexity.

The reason we don't use the release branches right now is because we've had a bad experience with them. UAT process can sometimes take months. Sometimes they request features, unreasonably, and upper management allows it. We've had our release branches become numerous, long-lived, with huge parity issues (in other words, they accumulate tons of changes over their lifetime).

This whole thing is pretty chaotic and tough to manage. However, QA gives us a ton of great feedback on this process because our releases are super stable. This is because QA touches every branch before it is merged, whereas before developers merged all day long to develop and QA couldn't keep up with the changes, so a bunch of stuff went untested in releases.

Now that I've explained our process and situation, some questions:

  1. How can we simplify this process?
  2. How should release branches be handled when you have UAT process in the middle? Ideally we want to be able to continue active development on develop after UAT starts.
  • If your own QA is your bottleneck, it seems pretty obvious where your problem is. If your customers QA is the bottleneck, you either have to talk with the customer or slow down your development for this customer. – Doc Brown Sep 2 '15 at 14:33
  • Sure, but you didn't address the release branch concern – void.pointer Sep 2 '15 at 15:31
3

Many familiar issues that I've also seen at many organizations.

My suggestions are as follows:

  • Branch off master for feature branches
  • Have QA do the merge back into master and push it
  • Release master weekly and tag each release with the release date

That's code and workflow-wise.

The harder parts are likely to be:

  • changing the culture
    There are probably a diversity of opinion on what is wrong and why it is. You'll need to have buy in and championing from the highest (relevant) level of the organization. You'll need to explain and sell the advantages of such a slimmed down system. Explain the simplicity and the rapid turnaround. Make sure your (probably) agile process is not just lip service but fully thought out from actual agile practices and, when possible, scrum masters.

  • changing the existing workflow You'll need to put together the new workflow and then develop a plan that will let you transition to it.

Other thoughts:

  • consider implementing a comprehensive feature switch system that will let your merge in (some) changes but have the features turns on/off through an admin gui.

  • in your deliver pipeline rather than finishing the feature and passing to QA, consider involving QA through the design, implementation and testing so that their final ok is a pass thru of well understood and agreed on assumptions. You don't want QA surprised by an implementation and you don't want developers surprise by how QA decides to test.

  • Have a weekly release meeting with all parties involved and go thru all the tickets. This can easily be 50-100 tickets. Touch base on each one for: release implications to the business, pre release steps, post release steps, downtimes, etc.

  • 1
    We tried having QA merge, but with submodules it became too much of a burden for them. Plus the developer may want to rebase and such prior to merging, and QA wouldn't have the domain expertise to resolve conflicts. Also we do employ feature toggles but not in the truest sense: There is still inherent risk of regressions due to legacy code and common libraries when code is changed for said feature. – void.pointer Sep 3 '15 at 16:01
  • submodules - arggh. They've been a major pain for several organizations that I have worked in. My last one reversed that decision eventually. The non-standard aspects of them and the need to manually performs updates, explain it to new people, etc eventually became too burdensome. We went to favoring simpliity and 1 method of how things work. – Michael Durrant Sep 4 '15 at 11:37
  • btw you can train the QA's to resolve the conflicts. I am QA and I resolve code merge conflicts. This frequently means grabbing the original developer for a few seconds (usually) and saying - is this what we want? 90% of the time you are good to go. In cases where the conflict is involve and complicated I pass it back to the developer to do the rebasen and get it right. This combines guidelines and practicality. Otherwise tickets can get 'lost' in the shuffle and take hours or days to complete when a quick conversation could resolve it – Michael Durrant Sep 4 '15 at 11:40
  • 1
    Our QA people are not programmers and probably never will be. That won't be a viable solution. Also submodules are required in Git to avoid monolithic repositories. Sparse checkouts are not available like they are in SVN. Kind of beside the point, though. – void.pointer Sep 14 '15 at 14:40
0

Ultimately we decided to go with a release branch per physical release out the door, regardless of how the customer intended to use it. If a released build happens to never make it through UAT, that's OK, we'll create another release branch and ship again. However, if UAT does accept the build, we do not get an opportunity to build it again. In that case they keep the build they got and we move on with the next release that may not be related to UAT at all.

In this respect, all of our releases follow the exact same flow and rules. I suspected we would be creating several release branches a week during UAT cycles but that is not the case. Our QA teams spend sufficient time on a release branch that we're only creating one every 2+ weeks.

So far this is working out great and is very manageable. It also forces feature releases to wait on UAT releases to complete, which is great. We can handle releases at a more granular level.

  • Dang this is one of those cases where I look back at one of my own answers 3 years later and don't like it. Hmm.. hopefully this question gets more traction in the future. I still don't feel comfortable with any of the answers, including my own. Maybe it's an unsolvable problem without the company making drastic changes to how they do business. – void.pointer Jun 4 '18 at 14:09
0

I know this question is a bit old, but we also went through a similar phase where things did not get merged to develop until it passed manual qa and hope my experience will help someone who is facing a similar problem.

The biggest issue is branches lying in isolation for too long and creating merge hells when you try to merge them back to develop and the developers having to find workarounds when a subsequent feature depends on the codes from an earlier branch which is not merged to develop yet.

So after wasting hours on end trying to merge old branches, we made the decision that developers will merge directly to develop when they are done and any bugs found on develop will be fixed as a matter of priority.

It's more of a cultural change as well since it puts the onus (correctly) on the developer to not to release anything until he has tested it and the qa s not being too happy about losing their 'gatekeeper authority'.

But we have been merging directly to develop for a while and never ever looked back.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.