Is it reasonable for DevOps to insist on 2 branch only integration?

Question

I am the DevOps/CI for my team of somewhere around 10 developers. We develop and implement a sophisticated scientific data collection and visualization web application. I am the person who introduced CI to my team and standardized and streamlined the build process using Jenkins, Maven, Python etc. We use Subversion for versioning, which is not my preference (of course, Git is) but I can work within those parameters. The subject of this inquiry is our branching strategy, which I am trying to promote into a policy.

As someone who manages all of the lifecycle past the point of checking changes in, my preference is to have not have to support more than two branches of code: (1-PROD) that which is currently deployed in PROD and (2-DEV) that which is in development. This simple scenario makes my life easy and the lack of flexibility with regard to branches is what enables efficiency and smooth operation in my department. Having a single, unified branch per release, and not having feature branches, eliminates the need for merging, which is something that is always a crapshoot, to put it mildly. So, the model I am proposing would, in theory, require to never merge anything -- if a developer makes PROD fix changes in 1-PROD, (s)he is responsible to apply those changes to 2-DEV, so that we don't have to do merges in the CI process. Keeps things neat and stable.

However, there is a tendency within the team as well as the management to resort to creating a new branch whenever the slightest discrepancy in requirements arises, and many of those challenges could be addressed within the same branch, e.g. by toggle switching the pending functionality using configuration until it is ready etc. In every instance of a new branch that has been made, I demonstrated that the isolation of the development effort could be accomplished within the existing branch using configuration settings. I wouldn't hate is as much if they didn't also require that official build artifacts be generated and deployed to various testing instances from those branches (PROD deployment always goes out from the main branch after any possible feature branch has been integrated). So if they want to have 100 branches for developers, that's fine by me as long as they are responsible for merging them into one of the two branches that affect me and I don't have to support deployment from there to actual instances. I would also hate it less if multiple branches didn't involve the awful process of merging, which I absolutely hate and strive to design an environment in which it's not done.

Is my proposed branching/deployment strategy (to support only the two canonical branches in the integration process) unreasonably simplistic? Is the complication/entanglement associated with branching flexibility simply a fact of life in the world of DevOps/CI?

Answer 1

The role of DevOps is to enable self-sufficiency among developers, not to take on all of the work yourself.

I believe you are right in encouraging developers to use feature toggles but unreasonable in limiting them to only two branches.

Feature toggles work well for new features that don't overlap with other features, but you'll eventually get into a situation where you have to overhaul something critical. Creating a new branch is an easy way to allow for major changes.

If you're using Jenkins, then it shouldn't be too much trouble to build and test any amount of branches and deploy the artifacts to testing environments. One benefit of this is the ability to notify developers when their branch fails to build and prevent them from breaking the dev branch.

In my previous role, I was able to set up a Jenkins instance which handled hundreds of builds per day from many branches, and developers/QA could use jobs to deploy the artifacts wherever they wanted. The point is that they were responsible for doing the work, I simply gave them the tools they needed.

Answer 2

This simple scenario makes my life easy and the lack of flexibility with regard to branches is what enables efficiency and smooth operation in my department.

If this is running flawlessly, that suggests to me that the features that are being committed are not conflicting with each other.

If the two features do not conflict, then you wouldn't encounter merge issue from a branch anyway.

If the two features do conflict, then you would have to handle merges in either case (branch merge, or resolve an update conflict which SVN enforces you to do before you get to commit your code).

---

the need for merging, which is something that is always a crap chute, to put it mildly

The issue here is one of perception.

Without extra branches, you indeed don't have to worry about merge issues, but you instead run into the continual issue of developer's breaking the branch and blocking everyone else. This can happen at any time, pretty much daily if you enforce a policy of committing code at the end of the day.

With extra branches, you actually defer those conflicts until it's actually time to merge.

While merge issues are more complicated to deal with, they also occur on your terms (only when you want to merge), rather than every commit being liable to break the build for everyone.

When you total the amount of effort spent, you'll see that the many little commit issues will take more time (especially when it affects multiple developers) compared to a single developer having to spend some effort to merge the branches.

---

Secondly, there is an inherent conflict with enforcing a policy of daily commits. What if a developer is working on a major change (one that changes things in a less than trivial way) which likely takes several days to complete?

• Should this developer ignore the policy to commit daily? That's bad, because it becomes a slippery slope of people deciding for themselves when they feel ready to commit code.
• Should the developer comment his changes and commit the application in a way that it works the way it used to? That takes additional effort, and it opens the door to forgetting to uncomment one or two parts of the changes and thus running into unexpected behavior. And that's even assuming that the changes don't drastically effect the classes/files because they it becomes impossible to manage.
• Should the developer commit the code as is, even if it's unfinished and therefore will render the application unusable until the developer has finished his task?

There's no good option here. But if you use feature branches, then you compartmentalize the issue. Whoever is working on the breaking change can do so at their own pace, in their own branch. Only when they are finished (i.e. they have working code), they merge it into the "shared" code.

Furthermore, there are ways to minimize the impact. Have your feature developers merge from the truck to their feature branch, and solve the merge conflicts that way. This dramatically lowers the emergence of merge conflicts when merging from the feature branch to the trunk.

Also, I would make it so the merging of a feature branch is the responsibility of the developer of that feature branch. You're actually already applying this principle; when you share a trunk, SVN forces you to update and resolve any potential conflicts before it allows you to commit.

You can enfore the same policy for your feature branches; where developers are asked to first merge the trunk to the branch, resolve the conflicts there (on their own pace, without affecting others), and then merging from the branch to the trunk.
It's still possible that there has been a new commit to the trunk between the two merge operations, but the odds are low, and even if conflicts occur, they will be relatively small (and thus not as prone to merge errors).

Answer 3

Yes, your strategy seems fine. With the following caveats.

1. Devs can still make as many "non building in CI" feature branches as they like.

2. You add a 3rd "Test" CI branch to allow dev of 1.2 to start while testing of 1.1 is in progress and 1.0 is still live and potentially needs hotfixes.

Having CI work for any branch can be done, but really you need a seperate envrionment to deploy that branch to. which again, in these days of cloud, can be done. But gets expensive fast and is generally much harder to achieve.

I think most places will have a live, dev and test envrionment setup and work around that. Forcing a release strategy and time table where feature releases have a degree of inflexiblity in the order in which they can be released.

However, you must realise that this is essentially a management and control issue about how people work. Not a simple techincal process. The best techincal solution is not always the selected solution in these cases.