Any advice for Git workflow using branch per development team (and not per feature)

Question

In our project, we have 5 teams working on a monolithic application in 3-week sprints. Until now, all commits go into master (via Gerrit). Our test pipeline is too lengthy to ensure quality pre-commit, so we have to accept that some commits break things. I'm aware that the root cause here is the pipeline, and a breaking commit should never reach master. We have to accept this, however, until our pipeline redesign is complete, and this will take several months in the least.

We're now discussing the approach of "team branches", where each team would develop and commit on their own branch, and update the master periodically. The heated part of the discussion revolves around merge or rebase. Our current understanding is:

Rebase

1. On branch-team1: git rebase origin/master
2. Resolve conflicts
3. Push to branch: git push origin branch-team1 --force-with-lease
4. Finally, rebase master: git rebase branch-team1

With this approach we see commit IDs changing (due to rebase + push --force), and part of our change history is invisible to Gerrit. The former has the effect that the entire team needs to halt work during the rebase, as their un-pushed work might get messed up due to the changing commit history.

On the plus side, this approach yields an easily readable history on master. Each commit is visible, and can be cherry picked or reverted.

Merge

1. On master: git merge origin/branch-team1 (+ resolve conflicts)
2. On branch-team1: git rebase origin/master

The main concern expressed here is that the history becomes much harder to read, as the master will mainly contain large "composite merge commits". For debugging/blaming, the associated team branch would have to be checked for the further history. Also, reverting a single commit (originally on a team branch) becomes more tedious.

However, a short proof-of-concept indicates that the history is no different to the Rebase approach, except for the actual merge commit. Similarly, cherry picks / reverts seem just as easily done, with the sole exception being if part of the cherry pick / revert would actually be a change made during conflict resolution and thus part of the merge commit. This, however, seems a tad far-fetched.

A third way?

It seems both approaches have negative aspects (personally I don't see much negative in the merge approach, but others on our project are clearly against it). It's a bit hard to believe that with the power and popularity of GIT, there isn't an established approach available which doesn't require squabbling over.

Does anyone have some practical experience to share here?

Answer 1

If you had to say one way which was 'standard' branching model for git it would be gitflow.

Many people find its not for them, but don't try to invent a new model until you've tried it.

Rebasing is bad, it deletes your commits and screws over anyone else using the branch. Never rebase. Especially never rebase if the only thing you don't like about merging is the way your source control browser presents the history of commits. Most tools offer various ways to display the history, exclude merge commits, only merge commits etc etc and the way the tool displays the history has no effect on the finished product.

Cherry picking is bad, it tricks you into thinking you can take random changes to your code base and expect them to compile out of sequence with the other changes. Never cherry pick.

The key thing with any branching model is to try and minimise difficult merges. One of the key causes of these is long lived branches. The longer a branch is being worked on without being merged the higher the chance that you will get a conflict when you come to merge it.

Your team branching model would result in lots of long lived branches, each written by people who are not working together and don't understand each others changes.

Your sprint is too long. Reduce it to 1 week sprints. This will increase the frequency and decrease the size of merges.

Your monolith is too monolith. Split it into smaller components and move each component onto its own git repo/build chain. This will decrease the number of people working on the same repo and thus merges

Answer 2

First, if Gerrit loses some changes on "external" branch rebasing, it looks fairly natural due to its approach to keep a branch per change: with rebasing of basic branch, these changes start hanging on thin air. Gerrit is nearly the best Git server, but it was not designed for such actions: once a commit is submitted to a branch, it shall never change. You may, of course, develop a tool which updates all proposed commit branches like refs/changes/56/123456/3... if this is feasible (well, work amount is huge, but what if some rebase fails?)

So, the only approach which looks proper with Gerrit is to use only merges and not rebases of submitted commits. Otherwise, you should use another server (Gitlab, BitBucket, etc.), losing all Gerrit merits but allowing freedom of rebases.

A possible (and used) manner with Gerrit is to allow proposing merge commits from users. Our neighbor department actively uses it. With this manner, merge commits are sendable but merge by Gerrit itself is prohibited. You should merge parent (master, develop...) branch to inferior (per-team) one often enough to minimize merge conflicts, solve conflicts locally, and propose a working version. After this, similarly, merge team branch to common parent. If needed, an automated external checker should be configured to check requirements (setting Verified flag). This will look similar to "gitflow" but with Gerrit advantages.