$DAYJOB, we have the following release/delivery pipeline:
- Get new features, work on new features.
- Once all features are completed, our own QA performs regressions, etc.
- QA signs off on first UAT build, ships to customer UAT department (this is really customer's own QA force). This begins our initial version (e.g. v1.0)
- Customer QA (UAT) finds issues, sends us back a list.
- Development addresses the list of issues. Sometimes they might ask for non-defects, such as improvements or small features. We triage these on a case-by-case basis. Our own QA also verifies the fixes. They sign off, we ship a new build (this gets a version bump, like v1.1)
- This "get list" and "work on the list" process cycles a few times, until customer signs off on the build (UAT Pass).
- Signed-off build becomes "production". Now developers can start working on new features for the next release.
Some notes on this process:
- We use a bastardized version of git-flow for this process (will explain in more detail later).
- Regarding versioning, we do not differentiate between UAT releases and production releases. We have to assume any release to UAT might be signed off and go gold at any time. For us, each build "out the door" is treated as a "production release".
- Developers work on branches off of
developfor new features. We even work on features for the next release. However, we do not merge features until they are scheduled for release. This is because we do not create 'release' branches (more on this later).
Here is how our development process goes for both bugs and features:
- Developer branches off
developand begins work.
- Developer finishes work, performs code review
- Code review passes, and we indicate to QA the branch is ready for testing. Our CI build system builds every branch in the remote repository, and QA can grab ad-hoc builds off of any branch. QA tests features in branches (and not off of
develop) for a few reasons:
- Builds on
developmight include multiple features or bug fixes from multiple developers over the course of some time. When regressions occur, it becomes more difficult to track down what caused the regressions. The "test-each-branch" model allows us to isolate regressions to the changes on that branch (i.e. the one feature or bugfix).
- All features must be tested on a branch because we do not merge them into
developuntil QA is happy with it. Also QA controls deployment schedule, so they know when develop is ready for new features.
- We're tight on QA resources. At the moment we have a 4-to-1 ratio of developers to QA personnel. They simply can't keep up with the work we do.
- Builds on
- Once QA is happy with the feature on the branch, they mark it as "verified".
- Once verified, it sits there unchanged until someone with appropriate authority "approves" the feature or branch. At this point, a developer rebases/merges latest into the branch, fixes conflicts, then merges it into
When we finally release software (new UAT build usually), we do a merge from
develop directly into
Symptoms of the current system:
- Tons of branches sitting around waiting to be merged. Over 100 at the moment, and the turnaround time is sometimes months before they are merged.
- Process is complex and tightly controlled by QA
- Throw submodules into the mix and you've got a recipe for disaster and complexity.
The reason we don't use the
release branches right now is because we've had a bad experience with them. UAT process can sometimes take months. Sometimes they request features, unreasonably, and upper management allows it. We've had our release branches become numerous, long-lived, with huge parity issues (in other words, they accumulate tons of changes over their lifetime).
This whole thing is pretty chaotic and tough to manage. However, QA gives us a ton of great feedback on this process because our releases are super stable. This is because QA touches every branch before it is merged, whereas before developers merged all day long to
develop and QA couldn't keep up with the changes, so a bunch of stuff went untested in releases.
Now that I've explained our process and situation, some questions:
- How can we simplify this process?
- How should
releasebranches be handled when you have UAT process in the middle? Ideally we want to be able to continue active development on
developafter UAT starts.