I am yet another Subversion user struggling to re-educate myself in the Tao of distributed version control.

When using Subversion, I was a big fan of the project-minor approach, and, with most of my former employers, we would structure our repository branches; tags & trunk as follows:

         |          +-alice-+
         |          |       +-shinyNewFeature
         |          |       +-AUTOMATED-+
         |          |                   +-shinyNewFeature
         |          +-bob-+
         |                +-AUTOMATED-+
         |                            +-bespokeCustomerProject

Within the actual source tree itself, we would use (something like) the following structure:

        |                       +-testAutomation
        |                       +-deploymentAutomation
        |                       +-docGeneration
        |                       +-staticAnalysis
        |                       +-systemTest
        |                       +-performanceMeasurement
        |                       +-configurationManagement
        |                       +-utilities
        |           +-log-+
        |           |     +-build
        |           |     +-doc
        |           |     +-test
        |           +-statistics-+
        |           |            +-build
        |           |            +-doc
        |           |            +-test
        |           +-charting-+
        |           |          +-build
        |           |          +-doc
        |           |          +-test
        |           +-distributedComputing-+
        |           |                      +-build
        |           |                      +-doc
        |           |                      +-test
        |           +-widgets-+
        |                     +-build
        |                     +-doc
        |                     +-test
        |              +-flagshipProduct-+
        |              |                 +-coolFeature
        |              |                 +-anotherCoolFeature
        |              |                 +-build
        |              |                 +-doc
        |              |                 +-test
        |              +-coolNewProduct
                  |                      +-bespokeProjectOne
                  |                      +-bespokeProjectTwo

The idea was (and still is) to use the structure of the repository to help structure communication between the engineering team; the customer-facing part of the business and various other stakeholders & domain experts.

To wit: Source documents that sit in one of the "project" directories get used (and earn money) only once. Documents that sit in one of the "productLines" directories earn money as many times as a product from that particular line gets sold. Documents that sit in one of the "libraries" directories earn money as many times as any of the products that use them get sold.

It makes the notion of amortization of costs explicit, and helps build support for source document reuse across the business.

It also means that there is a common structure over which our build automation tools can operate. (Our build scripts walk the source tree looking for "build" folders within which they find configuration files specifying how each component is to be built; a similar process happens for documentation generation and testing).

Significantly, the products on which I work typically take a LONG time to run performance measurement & characterization tests; from 20 to 200 hours; generating somewhere between several GB to several TB of processed test results/intermediate data (that must be stored and tied to a particular system configuration so performance improvement over time can be measured). This issue makes configuration management an important consideration, and also imposes some requirement for centralisation, as typically the computational resources needed to run the performance measurement and characterization tests are limited; (a small cluster of 64-128 cores).

As one final note; the continuous integration system knows that it needs to trigger a build; static analysis; smoke test & unit test run each time trunk is modified, each time any "tag" branch is modified, and each time any "AUTOMATED" branch branch is modified. This way, individual developers can use the CI system with their personal branches, an important capability, IMHO.

Now, here is my question: How can I replicate all of the above (and improve upon it, if possible), with Mercurial.


My current line of thinking is to use a central Subversion Repository, to define the overall structure, but to allow the use of hg as a client so developers can have repos available locally.

  • 1
    Wow. A good answer to this is going to be a very long essay I think.
    – Ed James
    Jan 4 '12 at 17:25
  • I think the key question is how and where are code merges going as that probably will define the path of least resistance. So, how does the code get merged? Jan 4 '12 at 18:17
  • Typically, a merge might come from a personal branch into a project or feature branch, and then into trunk. I never experienced too many difficulties with merges (we were using TortoiseSVN on Win32), although we never ran for too long (one iteration at most) without integrating back into trunk. We tended to do most of our work in trunk anyway, although the goal was to simplify the man-management rather than the development workflow. (One dev-lead, many independently working developers, so having everything in trunk made it easier on the dev-lead to keep track of what was happening.) Jan 4 '12 at 19:10
  • One key point was a heavy reliance on testing driven by the CI system, particularly at the system-test level. This was to help build confidence that different developers were not interfering with one another, and to promote a many-small-iterations mentality. (Also, the computational heavy-lifting required to run the system tests meant that there was less contention for computational resources if people worked mainly on trunk). Jan 4 '12 at 19:16

Spoike's answer is excellent, but there are a few things I think it would be worth adding which are too large for comments.

Branch organisation

With Mercurial you can happily ignore the whole of your first organisational chart. As Spoke says, each repository has it's own set of tags, branches (named and anonymous) and can be organised according to business need.

If bespokeProjectTwo needs a special version of the charting library, then you would branch charting, add the new facilities and use it in bespokeProjectTwo. The new facilities (and their bugs) would not be used by other projects which would reference the standard charting library. If the main charting library had bugs fixed, you could merge those changes into the branch. If other projects also needed these facilities, you could either get those projects to use the special branch, or merge the branch up into the main-line and close the branch.

Also, there is nothing stopping you having a policy to structure branch names to provide specific facilities like your AUTOMATION branches.

Directory organisation

There is no reason why you can't keep your source directory exactly as it is with Mercurial. The only difference is that whereas with Subversion you have a single monolithic (src) repository, with Mercurial you are better off splitting into repositories which are logically grouped. From your source tree structure, I would probably extract out each of the following as individual repositories:

      |           +-(log)
      |           +-(statistics)
      |           +-(charting)
      |           +-(distributedComputing)
      |           +-(widgets)
      |              +-(flagshipProduct)
      |              +-(coolNewProduct)
                |                      +-(bespokeProjectOne)
                |                      +-(bespokeProjectTwo)

This allows any product or bespoke project to use any combination of libraries, at any revision. Have a look at mercurial sub-repositories for an easy way to manage which libraries are used for any given version of a product or project.


An alternative to Spoike's suggested workflow (developer pulls from blessed repo, works locally, issues a pull request and finally the integrator pulls those changes & merges them) would be to use the continuous integration system as an intermediary.

As before, the developer pulls from blessed repo and works locally, but when done, they pull from the blessed repo again and merge themselves before pushing to an unblessed repo. Any changes to the unblessed repo are then reviewed (either manually or automatically) and moved to the blessed repo only if they are approved.

This means that the integrator only has accept or reject a change, not do the merge. In my experience it is almost always better for the developer who wrote the code to perform the merge than for someone else to do it.

As suggested in the mercurial book, hooks can be used to automate this procedure:

When someone pushes a changeset to the server that everyone pulls from, the server will test the changeset before it accepts it as permanent, and reject it if it fails to pass the test suite. If people only pull changes from this filtering server, it will serve to ensure that all changes that people pull have been automatically vetted.

Other issues

The problem of large test datasets can also be solved by putting that test data into a mercurial sub-repository. This will prevent the code repository getting bloated with test data, while still keeping the test data under revision control.

  • Again, another excellent and informative answer. Thank you. Jan 5 '12 at 14:37
  • RE: Branch Organisation. I agree that the first organizational chart can happily be ignored. It did not communicate the workflow particularly well, anyway, and so was not providing any real utility beyond reinforcing convention. I would like to replace it, however, with something that strongly communicates a (simple-as-possible) workflow, and encourages frequent commits. Perhaps calling the main "trunk/development" branch "daily" would do that? Jan 5 '12 at 14:41
  • RE: Directory Organisation. I was using the source directory organisation as a subliminal means of communication; imposing an implicit structure on the organisation of the code (and through that on the business as a whole). I am beginning to understand that Mercurial tends to be used in a very very flexible way; but I really want to constrain some of that flexibility to impose a structure on the way that people think about the business by imposing a structure on the way that their documents are organized on their workstations and our network storage areas. (More corporate comms than tech.) Jan 5 '12 at 14:48
  • RE: Workflow. I think that the simplest workflow would be to pull from a "daily" repository, work on it locally, then (frequently) push back to the "daily" repository, kicking off static analysis, smoke tests & regression tests via the CI system. I am happy for the main repo to be "broken", as long as I know about it, and as long as it gets fixed again quickly. In fact, I am considering making committing to the "daily" repo the only way that one can compile & build, to encourage frequent commits & good test coverage. (Far more important than the ability to work in isolation, IMHO). Jan 5 '12 at 14:51
  • @WilliamPayne - Thanks. While Mercurial is flexible, with appropriate repositories, branches and hooks you can build in whatever restrictions you want, at the organisational or repository level. Personally, I would start simply with organisational controls, and a few CI hooks, and extend those controls in the future as their need becomes apparent. Also, judicious use of sub-repos could, for example, encourage people check things out locally in the same structure as it is on the server, for instance by having productLines or bigImportantCustomer as super-repos.
    – Mark Booth
    Jan 5 '12 at 14:54

Okay, trying to answer this simply.

What you need to know

First thing you need to know: Mercurial is distributed version control and has some properties you should know about listed below.

  • The source come from one repository, where that repository can be cloned. All cloned repositories can share code with each other through synching (with pull and push commands, that can be access restricted).
  • Every user that has a copy of the code, has a clone of the repository. If they want to branch, they can do it in their local clone. That means you don't need to organize how every user should branch. They can do this for themselves.
  • Tags are created in mercurial by a commit (which is the same as hard tags in git). This means you don't need a directory inside your repository structure for tags.
  • The usual model that people work with in DVCS (which is employed in github and bitbucket) is to do it semi-centralized.

    Each user has a public repository (in some share or on a secure server) and a private repository (in their own workstations). They are both clones of an integrator's "blessed" repository. Whenever they feel they're ready to publish their code, they can push the changes from to their public repository. An integrator can then choose which users to pull code into the "blessed" repository.

    If the integrator can't merge some user's code easily, then the changes are rejected and it is up to that particular user to update their repository and fix the merge themselves. It is usually not that difficult if you merge often (as it is less code that needs to be merged) and usually that user should know what went wrong with the merge.

Repositories-per-project setup

So the usual set up is that for each project there is the following:

  • A public read-only repository that the integrator is responsible for. It is "blessed".

    I.e. all users can pull/fetch content but have no access to push it.

  • Each user can have their own public clone of the repository.

    Easiest set up as put in a share drive (though you might consider hosting such as bitbucket). The integrator receives pull-requests from the users and tries to pull the new code from these repositories. When merges are done without a hitch, it is put in the read-only repository. If not, then users are asked to fix the merge conflicts that arise by updating and merging it for themselves locally.

  • Each user can have their own private clones of the repository.

    Good practice is to pull from their public clone, but it doesn't matter if they pull from their public or the integrator's. All commits are uniquely identifiable so merging commits that you forgot to fetch in the public one is relatively easy to fix (by pushing the changes from the private to the public, it automatically gets the integrator's changes as well).

Source code organization

As in how to arrange the project source itself is something you need to think through. If an artifact needs to be source controlled then put it in source control. Personally I don't like the idea of checking in artifacts that are made by the build or runtime (due to the high risk of merge conflicts on these kinds of artifacts) such as binaries or log files.

You can also check in configuration, as long as they make it easy for developers to get going and doesn't mess up configuration for releases or live/production environment (such as app/web server settings). This leads to the notion that if the configuration you have seriously hinders the developers to get started within five minutes after they've checked out the code then it needs to be refactored. Another requirements is that it should be darn hard for the developers to mess up the release or live/production environment.

You mention that you have test data that needs to be tied to some version of the code. Now this is a bit trickier because DVCS-systems such as Mercurial and Git have a tendency to get slow when you check in data that is HUGE. In my experience it gets really unbearable after 5 GB of binary files (your milage may vary, so you should test out how it works out for you). I would however recommend that you put generated data into it's own repository and have your test system tag them appropriately when checking them in (and/or create text files for the same meta data purposes).

I hope this all makes sense. Please comment below if I missed some detail or if something needs further explaining and I'll try to edit.

  • +1 for a very nice response with several very useful points. In response to the first section of your answer, I had not grasped the significance of each user having their own public repository. Perhaps I need to think more about how peer-to-peer workflows could be organized. Jan 4 '12 at 21:52
  • In response to the second section of your answer, the whole point (in my mind) of having a single repository for the whole organisation is to create a shared mental image for how work is structured, and to make it easier to find components that can be re-used. (Very much the Cathedral rather than the Bazaar, but that is the environment in which I work). I would really like to know how to achieve the same sense of structured organization (filing system) with a DCVS. Jan 4 '12 at 22:00
  • In response to the third section of your answer: I agree wholeheartedly that the source control system is for source documents, and derived artefacts do not belong there. I also agree that it is impractical to store large binaries of any description in VCS. I do, however, believe that you can store large binaries in an agreed network location, with a defined name, and reference them from within the VCS. For example, the build environment(s) can be stored as named VM disk images, and referenced from the various build scripts. (e.g: build me on build_env_A). The same thing holds for test data. Jan 4 '12 at 22:06
  • In the past, I have used a hierarchy of directories on a network drive, where the directory names are derived from the subversion revision number + hash of branch location to tie intermediate files & test results to particular revisions. This means that we have traceability without needing to store derived files in version control. Jan 4 '12 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.