How to maintain multiple release branches simulateously (git)

Question

This question has probably been discussed to death, but I don't feel like I have a good answer. I work at a company where customers are on different versions of our product. When a bug is found, we determine what was the oldest release of the system that was impacted. Developers then checkout the appropriate branch for that bug (or hotfix branch), make the fix, deploy it to a QA environment, verify the behavior, and then merge it forward to as many release branches as there are between that and master/main.

There's a lot of problems with this approach. Mostly, the closer you get to master/main, the more the codebase changes. Merging/rebasing/cherry-picking becomes a fool's errand, to the point where the developer essentially ends up rewriting the fix in a newer branch. QA obviously needs to test the fix in every release environment, then, too.

We also struggle to merge forward frequently enough. Since most developers on the team do not have merge rights, it falls on a more senior-level developer to merge forward. The typical result is multiple fixes will get merged forward at once. Meanwhile QA is blocked waiting for this to happen (even if it's for a few days). The senior developer also is not intimately familiar with all these changes so it can be a struggle to guess/coordinate on how the merge should take place, or whether to ask the original developer to just re-implement their changes.

Because doing a rebase would be difficult, often we do merges/cherry-picks instead. We also do a lot of squashing. The end result is that the git history is misleading, to say the least. For example, I look like a superstar because my name ends up being on almost every line of code because I did the merge.

Some companies seem to push the idea of fixing bugs in master/main, and then back-porting them instead. I am not sure that is much easier or more true to history.

Typically, throughout my career, when I struggled with stuff like this, it's been because the company I was working for was missing some fundamental idea or it was an organizational/practice-related problem. That said, my current company seems to be in a much better position that anywhere I've previously worked.

Answer 1

"This question has probably been discussed to death, but I don't feel like I have a good answer"

This is because the answer you would like is "here is a magic way the computer can automatically apply a human written fix to code A to similar code B" and we aren't that far along with the AI... yet.

If you have multiple versions of the code out there, then the default position is that you are going to have to manually fix them all individually. All this mucking about with merges isn't really helping you at all. You should probably stop it.

Here are your choices.

1. Refactor the system into modules. find a bug in a module, fix the bug and upgrade the module in all your released versions.

   Of course you still have to test each one and work out compatibility between all the module versions etc etc. But at least you don't have merges anymore.

2. Stop maintaining old versions. Tell you customers to upgrade.

This is what most people do. Get tough with your release schedule, decide what level of support you are going to commit to, have less supported versions. Maybe one a year with updates through the year. Support for only this and last years version?

Sure I can hear the objections already, the customers wont like it! they pay our bills! Well then hire dev/test/support teams per version and stop pretending that you can support all these versions for free.

Answer 2

This is indeed a hard problem.

The first step is to make sure that you aren't supporting all old versions of the product. Companies have supported versions for a reason - those are the ones that, if affected by a bug, would get fixed. If a customer has not upgraded, they may not get bug fixes. Some companies offer paid support for their products where a company can use an older version but pay for added support for that otherwise unsupported version.

You're also going to want robust automated tests. Building systems without a robust set of automated tests is risky. When a bug is reported, you can write the test and apply it to all the supported versions. For example, if you write a test that fails an assertion because of the bug and it either passes or has an unexpected failure case, you can do more investigation into those particular versions and why the defect may not be present or if it may be present in a different way. For the versions where the tests fail as expected, making that test (and the rest of the suite) pass confirms the fix. Relying on deployments to the QA environment and manual verification steps is slow and error prone.

You may want to look at why merging is so restricted. Understanding why this is the case and addressing those concerns would help make the process smoother. Although it very well could be because there are no safety nets, such as automated testing, that allow developers to move quickly with high levels of confidence.

Answer 3

Reframing (sorry)

Customers want old versions for three reasons:

• in new versions features are changed and these changes are incompatible with the customer's other software/business processes.
• customers do not want the hasle to install updates.
• the new version does not run on the customer's old platform.

If you decide to support these user stories (take the warnings of other answers into account before doing so), you can do without having multiple release branches/supported software versions.

Supporting old features/behaviour you can do with feature flags. The open-closed principle is your friend in this. Strategy pattern and DI will help. Once you change or deprecate a feature move it somewhere where it's not hindering development of the current feature set. Stale dependencies you move there as well.

Human inertia (the reluctance to change/install updates) is a though beast to battle. Being bothered by a bug might motivate customers to upgrade. We might help our customers by creating installers that easily allow keeping features at their current behaviour.

The third is a difficult one. Switching target platform often has a huge impact on the code. Merging a bugfix from a branch targetting one platform to a branch targetting an other platform will often require recreating the fix anyway. So having a branch (or repo) per platform does not cause more work than it did before. But perhaps you and your team are heroes of insulation and you manage to switch platform via a feature flag.

Note: even if you ease development in this way, you still have to test all the variants and your support team has to intimately know all the variants.