I am the DevOps/CI for my team of somewhere around 10 developers. We develop and implement a sophisticated scientific data collection and visualization web application. I am the person who introduced CI to my team and standardized and streamlined the build process using Jenkins, Maven, Python etc. We use Subversion for versioning, which is not my preference (of course, Git is) but I can work within those parameters. The subject of this inquiry is our branching strategy, which I am trying to promote into a policy.

As someone who manages all of the lifecycle past the point of checking changes in, my preference is to have not have to support more than two branches of code: (1-PROD) that which is currently deployed in PROD and (2-DEV) that which is in development. This simple scenario makes my life easy and the lack of flexibility with regard to branches is what enables efficiency and smooth operation in my department. Having a single, unified branch per release, and not having feature branches, eliminates the need for merging, which is something that is always a crapshoot, to put it mildly. So, the model I am proposing would, in theory, require to never merge anything -- if a developer makes PROD fix changes in 1-PROD, (s)he is responsible to apply those changes to 2-DEV, so that we don't have to do merges in the CI process. Keeps things neat and stable.

However, there is a tendency within the team as well as the management to resort to creating a new branch whenever the slightest discrepancy in requirements arises, and many of those challenges could be addressed within the same branch, e.g. by toggle switching the pending functionality using configuration until it is ready etc. In every instance of a new branch that has been made, I demonstrated that the isolation of the development effort could be accomplished within the existing branch using configuration settings. I wouldn't hate is as much if they didn't also require that official build artifacts be generated and deployed to various testing instances from those branches (PROD deployment always goes out from the main branch after any possible feature branch has been integrated). So if they want to have 100 branches for developers, that's fine by me as long as they are responsible for merging them into one of the two branches that affect me and I don't have to support deployment from there to actual instances. I would also hate it less if multiple branches didn't involve the awful process of merging, which I absolutely hate and strive to design an environment in which it's not done.

Is my proposed branching/deployment strategy (to support only the two canonical branches in the integration process) unreasonably simplistic? Is the complication/entanglement associated with branching flexibility simply a fact of life in the world of DevOps/CI?

  • If someone makes a fix in your production branch, you want them to manually repeat it into the main branch, without merging? That just seems bizarre. I use SVN daily, and merging for the most part is not a big deal. Maybe you just learn to become more comfortable doing it. – Andy Oct 2 '18 at 23:00
  • 2
    "We use Subversion for versioning, which is not my preference (of course, Git is) " WHY? You want to work with only two branches, with everyone commiting on trunk (DEV) and one release branch (PROD)? Why would you want to use Git like SVN? – Kevin Van Dyck Oct 3 '18 at 7:04
  • It's easier to alternate between the branches in git and also easier to manage commits, such as delete them or compress them etc – amphibient Oct 3 '18 at 12:58
  • @Andy -- It's not that I want them to do it, it's that I want to engineer a process whereby I spare myself of doing it – amphibient Oct 3 '18 at 14:56
  • Can developes build and try out the project with their changes locally? – max630 Oct 4 '18 at 7:15

The role of DevOps is to enable self-sufficiency among developers, not to take on all of the work yourself.

I believe you are right in encouraging developers to use feature toggles but unreasonable in limiting them to only two branches.

Feature toggles work well for new features that don't overlap with other features, but you'll eventually get into a situation where you have to overhaul something critical. Creating a new branch is an easy way to allow for major changes.

If you're using Jenkins, then it shouldn't be too much trouble to build and test any amount of branches and deploy the artifacts to testing environments. One benefit of this is the ability to notify developers when their branch fails to build and prevent them from breaking the dev branch.

In my previous role, I was able to set up a Jenkins instance which handled hundreds of builds per day from many branches, and developers/QA could use jobs to deploy the artifacts wherever they wanted. The point is that they were responsible for doing the work, I simply gave them the tools they needed.

  • If you're using Jenkins, then it shouldn't be too much trouble to build and test any amount of branches and deploy the artifacts to testing environments. it involves separate jobs which do separate builds out of separate branches. It increases the amount of potential problems – amphibient Oct 2 '18 at 18:49
  • with the 2 branch model, it allows me to have two fixed build jobs: current and pending, and then I just go and flip the branch from which they are done once the PROD release takes place. Supporting more branches for build would involve creating ad hoc jobs -- high maintenance. yes, i am trying to support traffic that is more akin to railroad than highway where vehicle have to move on fixed rails vs being able to pass and move however they want – amphibient Oct 2 '18 at 18:53
  • 1
    Ideally it shouldn't involve separate build jobs, but rather a parameterized job which can build any branch. I used job templates extensively for this purpose. stackoverflow.com/a/32622892/3608792 – Dan Wilson Oct 2 '18 at 18:53
  • 1
    I avoid job parameterization because I want my Jenkins users to just have a one click build/deploy and not have to supply arguments – amphibient Oct 2 '18 at 18:54
  • 4
    @amphibient I believe you have cultural issues to solve, both with the team and yourself, if you really want DevOps to take hold. The whole point is getting you and the devs working more closely together to deliver value. It’s apparent from your attitude that you don’t trust the team, so why should they trust you? You have a skill they don’t. Your job is to teach that skill to others and why it’s an important skill to begin with. – RubberDuck Oct 2 '18 at 22:44

This simple scenario makes my life easy and the lack of flexibility with regard to branches is what enables efficiency and smooth operation in my department.

If this is running flawlessly, that suggests to me that the features that are being committed are not conflicting with each other.

If the two features do not conflict, then you wouldn't encounter merge issue from a branch anyway.

If the two features do conflict, then you would have to handle merges in either case (branch merge, or resolve an update conflict which SVN enforces you to do before you get to commit your code).

the need for merging, which is something that is always a crap chute, to put it mildly

The issue here is one of perception.

Without extra branches, you indeed don't have to worry about merge issues, but you instead run into the continual issue of developer's breaking the branch and blocking everyone else. This can happen at any time, pretty much daily if you enforce a policy of committing code at the end of the day.

With extra branches, you actually defer those conflicts until it's actually time to merge.

While merge issues are more complicated to deal with, they also occur on your terms (only when you want to merge), rather than every commit being liable to break the build for everyone.

When you total the amount of effort spent, you'll see that the many little commit issues will take more time (especially when it affects multiple developers) compared to a single developer having to spend some effort to merge the branches.

Secondly, there is an inherent conflict with enforcing a policy of daily commits. What if a developer is working on a major change (one that changes things in a less than trivial way) which likely takes several days to complete?

  • Should this developer ignore the policy to commit daily? That's bad, because it becomes a slippery slope of people deciding for themselves when they feel ready to commit code.
  • Should the developer comment his changes and commit the application in a way that it works the way it used to? That takes additional effort, and it opens the door to forgetting to uncomment one or two parts of the changes and thus running into unexpected behavior. And that's even assuming that the changes don't drastically effect the classes/files because they it becomes impossible to manage.
  • Should the developer commit the code as is, even if it's unfinished and therefore will render the application unusable until the developer has finished his task?

There's no good option here. But if you use feature branches, then you compartmentalize the issue. Whoever is working on the breaking change can do so at their own pace, in their own branch. Only when they are finished (i.e. they have working code), they merge it into the "shared" code.

Furthermore, there are ways to minimize the impact. Have your feature developers merge from the truck to their feature branch, and solve the merge conflicts that way. This dramatically lowers the emergence of merge conflicts when merging from the feature branch to the trunk.

Also, I would make it so the merging of a feature branch is the responsibility of the developer of that feature branch. You're actually already applying this principle; when you share a trunk, SVN forces you to update and resolve any potential conflicts before it allows you to commit.

You can enfore the same policy for your feature branches; where developers are asked to first merge the trunk to the branch, resolve the conflicts there (on their own pace, without affecting others), and then merging from the branch to the trunk.
It's still possible that there has been a new commit to the trunk between the two merge operations, but the odds are low, and even if conflicts occur, they will be relatively small (and thus not as prone to merge errors).

  • continual issue of developer's breaking the branch and blocking everyone else -- when that happens, I just go and take their checkin out altogether, which is made possible by continuous integration -- i find it out within an hour of when they break it so there is usually no changes built on top of what I need to take out – amphibient Oct 3 '18 at 14:59
  • you compartmentalize the issue -- yes, that is nice but the problem is that the final product is not compartmentalized -- it is amalgamated and you can test two compartmentalized branches and everything is peachy but if you merge them (which corresponds to the final product), you may get crap. So, while I recognize the nice aspects of compartmentalization, it often does not correspond with the ultimate reality. Things work as a whole... – amphibient Oct 3 '18 at 15:02
  • @amphibient: (first comment) If you're able to deal with checking that break the build; why would you then not be able to deal with merge errors that break the build? (second comment) Why would you not also test the final result after the merge? I'm not understanding why you're deciding to only test the separate branches and then merge and not test again. You say there's CI so retesting should be a given, since a merge counts as a commit, which should trigger a build, which includes running the tests. – Flater Oct 4 '18 at 17:52
  • @amphibient The point of compartmentatlization is not to test two features separately and, it is to develop features separately (you can then also test them separately, but that's a secondary benefit). Just think about developing two features simultaneously, both of them taking multiple days/weeks to implement (and therefore, during development of a feature, the application does not build until the featureis finished). Are you really going to let your developers develop days/weeks worth of code without checking in? A HDD crash is going to ruin your project planning/budget. – Flater Oct 4 '18 at 17:57
  • I never said I wouldn't test after the merge. I am trying to keep everything cleaner by not having to merge. – amphibient Oct 4 '18 at 17:57

Yes, your strategy seems fine. With the following caveats.

  1. Devs can still make as many "non building in CI" feature branches as they like.

  2. You add a 3rd "Test" CI branch to allow dev of 1.2 to start while testing of 1.1 is in progress and 1.0 is still live and potentially needs hotfixes.

Having CI work for any branch can be done, but really you need a seperate envrionment to deploy that branch to. which again, in these days of cloud, can be done. But gets expensive fast and is generally much harder to achieve.

I think most places will have a live, dev and test envrionment setup and work around that. Forcing a release strategy and time table where feature releases have a degree of inflexiblity in the order in which they can be released.

However, you must realise that this is essentially a management and control issue about how people work. Not a simple techincal process. The best techincal solution is not always the selected solution in these cases.

  • yes, I was already pre-emptively stipulating to your 1. I am not sure I understand what your Test CI branch would do that my 1-PROD and 2-DEV do not – amphibient Oct 2 '18 at 21:50
  • 1
    well, say im a dev merging in my features to dev as they are finished. If the test team is testing that branch, then I just broke their process. similarly if we merge dev to prod and test there, but then I need to merge in a hot fix for a live issue – Ewan Oct 2 '18 at 21:53
  • oh ok -- I am well familiar with that scenario. actually, before that scenario breaks the build in Jenkins -- the merge should not have been checked it because the dev responsible should have seen it break in their local env. When things like that happen, whether due to a merge of simple good ole laziness to build locally before checking in, i revert the change that broke the build out and send out a semi-nasty email to the team reminding them of the process – amphibient Oct 2 '18 at 21:56
  • But if the breakage is functional and not within the build -- still the same -- they should not have checked it in. – amphibient Oct 2 '18 at 21:56
  • 1
    ok, so what im saying is that you need a staging branch to go with your staging environment. This allows you to do bugs fixes on the version in staging, without introducing the new features that have been added to dev in the time between that version moving to stage and the bug being found. – Ewan Oct 2 '18 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.