Sorry about this long post, but I think it is worth it.

I have just started with a small .NET shop that operates quite a bit differently to other places that I have worked. Unlike any of my previous positions, the software written here is targeted at multiple customers and not every customer gets the latest release of the software at the same time. As such, there is no "current production version." When a customer does get an update, they also get all of the features added to the software since their last update, which could be a long time ago. The software is highly configurable and features can be turned on and off: so called "feature toggles." Release cycles are very tight here, in fact they are not on a schedule: when a feature is complete the software is deployed to the relevant customer.

The team only last year moved from Visual Source Safe to Team Foundation Server. The problem is they still use TFS as if it were VSS and enforce Checkout locks on a single code branch. Whenever a bug fix gets put out into the field (even for a single customer) they simply build whatever is in TFS, test the bug was fixed and deploy to the customer! (Myself coming from a pharma and medical devices software background this is unbelievable!). The result is that half baked dev code gets put into production without being even tested. Bugs are always slipping into release builds, but often a customer who just got a build will not see these bugs if they don't use the feature the bug is in. The director knows this is a problem as the company is starting to grow all of a sudden with some big clients coming on board and more smaller ones.

I have been asked to look at source control options in order to eliminate deploying of buggy or unfinished code but to not sacrifice the somewhat asynchronous nature of the teams releases. I have used VSS, TFS, SVN and Bazaar in my career, but TFS is where most of my experience has been.

Previously most teams I have worked with use a two or three branch solution of Dev-Test-Prod, where for a month developers work directly in Dev and then changes are merged to Test then Prod, or promoted "when its done" rather than on a fixed cycle. Automated builds were used, using either Cruise Control or Team Build. In my previous job Bazaar was used sitting on top of SVN: devs worked in their own small feature branches then pushed their changes to SVN (which was tied into TeamCity). This was nice in that it was easy to isolate changes and share them with other peoples branches.

With both of these models there was a central dev and prod (and sometimes test) branch through which code was pushed (and labels were used to mark builds in prod from which releases were made...and these were made into branches for bug fixes to releases and merged back to dev). This doesn't really suit the way of working here, however: there is no order to when various features will be released, they get pushed when they are complete.

With this requirement the "continuous integration" approach as I see it breaks down. To get a new feature out with continuous integration it has to be pushed via dev-test-prod and that will capture any unfinished work in dev.

I am thinking that to overcome this we should go down a heavily feature branched model with NO dev-test-prod branches, rather the source should exist as a series of feature branches which when development work is complete are locked, tested, fixed, locked, tested and then released. Other feature branches can grab changes from other branches when they need/want, so eventually all changes get absorbed into everyone elses. This fits very much down a pure Bazaar model from what I experienced at my last job.

As flexible as this sounds it just seems odd to not have a dev trunk or prod branch somewhere, and I am worried about branches forking never to re-integrate, or small late changes made that never get pulled across to other branches and developers complaining about merge disasters...

What are peoples thoughts on this?

A second final question: I am somewhat confused about the exact definition of distributed source control: some people seem to suggest it is about just not having a central repository like TFS or SVN, some say it is about being disconnected (SVN is 90% disconnected and TFS has a perfectly functional offline mode) and others say it is about Feature Branching and ease of merging between branches with no parent-child relationship (TFS also has baseless merging!). Perhaps this is a second question!


5 Answers 5


What defines a DVCS?

The distributed in DVCS means that every clone of a repository has all of the information required to commit, update, branch, merge or search for any revision in that repository, without ever touching a server. The only thing you can do offline in svn is actually edit the files - you need server access for almost all svn commands, including doing something as simple as grepping an svn log, so it is actually closer to 0% than to 90%!

Any authoritative central repository that you might set up in a DVCS workflow is just another clone, and the only time you need to interact with it are when you pull down other peoples updates or when you push your own changes so that other people can see them, pretty much everything else can be done off-line.

What branching model is appropriate?

I've been in the situation you are in right now. Such systems can be a real pain but you have to understand the pragmatic reasons they became like this and realise that they are not beyond redemption. Many tools have been developed which can help manage this sort of complexity.

First of all, whatever you do, don't show people the successful git branching model, it will just confuse them and turn them off. Instead, develop your own model that reflects your existing workflow, yet solves the problems with your existing workflow.

Some resources which you may want to consider include things like git submodules which will allow different customer releases to specify different combinations of customer configuration, application modules and libraries. Another option would be the use of a patch management system to apply customer/product specific patch queues.

Both of these options will provide much greater flexibility, transparency and safety than your current workflow, and could be easier to use than an even more complex branch only strategy. I certainly wish I'd has access to these tools when I was in your situation.

For more information on these options, see my answers to Strategy to use version control on a modular system, How to Use Subversion Repository Inside Git Repository? and Source/Version control for application used by multiple companies.

Ultimately, this is really something you will have to develop with the rest of the team. If you have the vision to propose something which works better then what you already have, and can get the buy-in of your fellow developers, then you will have a much easier time of it.

The most important thing is to show your colleagues how what you propose will make their lives easier. Once they are convinced, you stand a much better chance of getting management to throw away their investment in TFS and start using a model more appropriate to your working methods.

  • 1
    +1 for finding the successful git branching model that works for you
    – jcmeloni
    Commented Mar 19, 2012 at 13:28
  • 1
    +1 for "make their lives easier". This is the major motivator.
    – user1249
    Commented Mar 19, 2012 at 13:39

Firstly, DVCS is a red-herring for the problems you have - the version control tool in use is not the root of the issues that have to be resolved. It may be that there are aspects of DVCS solutions that are "better" than TFS but its not what needs to be fixed at this point.

You've identified that you need a workable branching structure that fits your organisation - I think you'll find you still have a trunk, when a feature is complete it gets merged back into the trunk and get closed off. There are some nice thoughts about how you implement common dependencies too.

You also need to get continuous integration working (no reason not to have an automated build for each and every active branch to give you confidence that you can build that branch and that it passes the relevant tests). I am uncomfortable where a commit (or at least a "push") doesn't trigger a build for me.

And you need to start in on automated testing at all levels but especially unit tests and integration tests to start to reduce the chances of new bugs escaping into the wild. This last is huge and something I'm still struggling with but its clear that once you know you can build things at all this will have the most value.

You need to combine this with making sure that your deployment packages come from you build server and that deployment is automated as far as possible (you should be able to go from build server artifact to live deployed code with minimal effort and minimal stress).

Hmm, I assumed that there is a nice well ordered issue tracking setup... you need that too and to be confident that it is being used properly. Ideally you want your live applications to feed back errors to this system (or for triage) automatically.

Lastly, don't try to solve all your problems at once - build and test would seem to me to be the places you should focus first.

  • There is a part of me that agrees that DVCS may be a red herring as well: I certainly agree the issue is with the process more than anything else. I think that continuous integration may be a stretch here and unit testing just isn't going to happen as it will be seen as too much work. The codebase isnt testable anyway as it is a tightly coupled monolithic system and everything is based on conrete types: no interfaces.
    – MrLane
    Commented Mar 20, 2012 at 22:51
  • @MrLane - I know it's going to be difficult to explain this to people, but since I started developing in a TDD way, I'm increasingly convinced that I don't have the time to not write tests.
    – Mark Booth
    Commented Mar 21, 2012 at 16:19

Second question is easier and shorter, so I'll try to start from it

DVCS is system, where no one "authoritative" source of code (except "on agreement", when used) and P2P-exchange of data is possible without additional tiers (personal, non-canonical definition)

On the topic the first question

I'm afraid, company have to reconstruct workflow and rethink about style in order to get "somehow managed and predictable code". I can't say about TFS (except personal opinion and feeling, that it is weak system in Version Control part /baseless merge is evil/), but for any VCS in your situation ("Product" is set of independent "Modules", each "Customer" get different "Products" - is this assumption correct?) I'll prefer split development of modules into separate branches, have Product as "Supermodule" (also branch?), there each module tied to specific revision of module-branch, module-development uses branch-per-task paradigm (and module-branch consist only from mergesets).

This way you can always know, which "Collection" (i.e set of modules and their corresponding revisions) forms each "Product", have possibility to do CI (for finished and merged task-branches), Unit-testing and Builds


Ad main question: I believe what you are talking about is exactly what git successful branching model (and the helper tool git flow to support it) is about. Have a master branch, that is always in deployable state and do all work on feature branches.

You can also use the process used by git itself, which is derived from the same basic principle. In git-core development, all work happens on feature branches. The feature branches are submitted to integrator, who runs a script to merge all of them to create a branch named pu (proposed updates). Various people take this branch and work with it to test it.

Instead of integrator, you can have continuous integration server do this merge at the begining of a build. That way whenever anybody of the team pushes changes to the central repository as a feature branch (probably using some naming convention to tell which branches should be selected).

In git the feature branch than proceeds to next, master or maint depending on which release it's targeting at (maint is for bugfixing current release, master for current in-preparation release and next for the one after that), but you won't have that many.

While the features are in pu ("cooking" in git maintainer's terminology), they are rewound and the pu branch is discarded and created again each time, which makes it easier to review, but unsuitable for basing other work on. When the feature branch is merged into one of the mainlines, it is closed for rewinds and further fixes are done as new commits.

I personally would recommend git best. It's a bit harder to learn initially, because it grows more organically, but in the end it seems most flexible. But any of the three distributed systems, git, mercurial and bazaar will serve you well (and you can often even mix them, e.g. mercurial can pull and push to/from git repository and I believe so can bazaar).

Ad second question: I was taught that "distributed", in general, means that you can move objects around and they keep their identity. Which is exactly what distributed version control does: you clone the repository and it contains the same commits and allows doing the same things with them. The ease of branching and the disconnected operation are the main user-level features that follow from the principle of moving commits around and the directed graph layout that allows it.


Unfortunately, there is no known solution to bugs in code :)

So you're just looking to stop unfinished checkins from being caught up in the main release, and the only answer to that is branch-work-merge for each developer. I did this in a previous company using Clearcase, it worked quite well (even though we had to have a dozen clearcase admins).

Now, I assume also that you perform bug fixing on the version of the product that each customer currently has... so you have a merge issue bringing bugfixes from version A all the way through to version Z. There's no easy way to deal with this, but you will have to have a branch for each shipped version. The best way to deal with this is to keep feature branches on the latest version only, and make customers upgrade to get the new features, at the same time, perform bugfixing directly on the release branch and merge them upwards to all the other release branches when complete.

Not too nice, but it can work remarkably well. You keep the code organised and nicely separate. Its also easy to understand for the other devs - small bugfixes directly on "the code", anything more than a couple of lines gets done on a dedicated branch where they can take as long as they like to complete it. (you will have to sort out the merge issues and reassure them that it's ok if 2 devs work on 2 features at the same time!!)

After a while you can introduce feature branches on the release branches too, where bugs are fixed then merged in, but IMHO this is typically more effort than needed. If you need to add features to old releases though, then you'll need to follow this approach - branch ff a release branch, then merge the code back onto that release, and merge the change upwards to the later releases too. That will make your test team very unhappy as releases will be heavily delayed due the need to test several versions repeatedly, and the dev team unhappy as they will have to do a lot of merging before they can start on new code (in my current company this happens, mainly due to the amount of work we have that always needs to be complete asap).


basically a DVCS is where everyone has their own copy of the server repo. It has some advantages (especially in distributed teams with limited commiuncation), but has some disadvantages too so check them out before you switch to a DVCS. If you're a Windows shop then you'll probably find Mercurial is the best DVCS for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.