Mercurial Repository structure with heavyweight corporate comms, configuration management & test requirements

Question

I am yet another Subversion user struggling to re-educate myself in the Tao of distributed version control.

When using Subversion, I was a big fan of the project-minor approach, and, with most of my former employers, we would structure our repository branches; tags & trunk as follows:

branches-+
         +-personal-+
         |          +-alice-+
         |          |       +-shinyNewFeature
         |          |       +-AUTOMATED-+
         |          |                   +-shinyNewFeature
         |          +-bob-+
         |                +-AUTOMATED-+
         |                            +-bespokeCustomerProject
         +-project-+
                   +-shinyNewFeature
                   +-fixStinkyBug
tags-+
     +-m20110401_releaseCandidate_0_1
     +-m20110505_release_0_1
     +-m20110602_milestone
trunk

Within the actual source tree itself, we would use (something like) the following structure:

  (src)-+
        +-developmentAutomation-+
        |                       +-testAutomation
        |                       +-deploymentAutomation
        |                       +-docGeneration
        |                       +-staticAnalysis
        |                       +-systemTest
        |                       +-performanceMeasurement
        |                       +-configurationManagement
        |                       +-utilities
        +-libraries-+
        |           +-log-+
        |           |     +-build
        |           |     +-doc
        |           |     +-test
        |           +-statistics-+
        |           |            +-build
        |           |            +-doc
        |           |            +-test
        |           +-charting-+
        |           |          +-build
        |           |          +-doc
        |           |          +-test
        |           +-distributedComputing-+
        |           |                      +-build
        |           |                      +-doc
        |           |                      +-test
        |           +-widgets-+
        |                     +-build
        |                     +-doc
        |                     +-test
        +-productLines-+
        |              +-flagshipProduct-+
        |              |                 +-coolFeature
        |              |                 +-anotherCoolFeature
        |              |                 +-build
        |              |                 +-doc
        |              |                 +-test
        |              +-coolNewProduct
        +-project-+
                  +-bigImportantCustomer-+
                  |                      +-bespokeProjectOne
                  |                      +-bespokeProjectTwo
                  +-anotherImportantCustomer-+
                                             +-anotherBespokeProject

The idea was (and still is) to use the structure of the repository to help structure communication between the engineering team; the customer-facing part of the business and various other stakeholders & domain experts.

To wit: Source documents that sit in one of the "project" directories get used (and earn money) only once. Documents that sit in one of the "productLines" directories earn money as many times as a product from that particular line gets sold. Documents that sit in one of the "libraries" directories earn money as many times as any of the products that use them get sold.

It makes the notion of amortization of costs explicit, and helps build support for source document reuse across the business.

It also means that there is a common structure over which our build automation tools can operate. (Our build scripts walk the source tree looking for "build" folders within which they find configuration files specifying how each component is to be built; a similar process happens for documentation generation and testing).

Significantly, the products on which I work typically take a LONG time to run performance measurement & characterization tests; from 20 to 200 hours; generating somewhere between several GB to several TB of processed test results/intermediate data (that must be stored and tied to a particular system configuration so performance improvement over time can be measured). This issue makes configuration management an important consideration, and also imposes some requirement for centralisation, as typically the computational resources needed to run the performance measurement and characterization tests are limited; (a small cluster of 64-128 cores).

As one final note; the continuous integration system knows that it needs to trigger a build; static analysis; smoke test & unit test run each time trunk is modified, each time any "tag" branch is modified, and each time any "AUTOMATED" branch branch is modified. This way, individual developers can use the CI system with their personal branches, an important capability, IMHO.

Now, here is my question: How can I replicate all of the above (and improve upon it, if possible), with Mercurial.

--edit:

My current line of thinking is to use a central Subversion Repository, to define the overall structure, but to allow the use of hg as a client so developers can have repos available locally.

Answer 1

Spoike's answer is excellent, but there are a few things I think it would be worth adding which are too large for comments.

Branch organisation

With Mercurial you can happily ignore the whole of your first organisational chart. As Spoke says, each repository has it's own set of tags, branches (named and anonymous) and can be organised according to business need.

If bespokeProjectTwo needs a special version of the charting library, then you would branch charting, add the new facilities and use it in bespokeProjectTwo. The new facilities (and their bugs) would not be used by other projects which would reference the standard charting library. If the main charting library had bugs fixed, you could merge those changes into the branch. If other projects also needed these facilities, you could either get those projects to use the special branch, or merge the branch up into the main-line and close the branch.

Also, there is nothing stopping you having a policy to structure branch names to provide specific facilities like your AUTOMATION branches.

Directory organisation

There is no reason why you can't keep your source directory exactly as it is with Mercurial. The only difference is that whereas with Subversion you have a single monolithic (src) repository, with Mercurial you are better off splitting into repositories which are logically grouped. From your source tree structure, I would probably extract out each of the following as individual repositories:

src-+
      +-(developmentAutomation)
      +-libraries-+
      |           +-(log)
      |           +-(statistics)
      |           +-(charting)
      |           +-(distributedComputing)
      |           +-(widgets)
      +-productLines-+
      |              +-(flagshipProduct)
      |              +-(coolNewProduct)
      +-project-+
                +-bigImportantCustomer-+
                |                      +-(bespokeProjectOne)
                |                      +-(bespokeProjectTwo)
                +-anotherImportantCustomer-+
                                           +-(anotherBespokeProject)

This allows any product or bespoke project to use any combination of libraries, at any revision. Have a look at mercurial sub-repositories for an easy way to manage which libraries are used for any given version of a product or project.

Workflow

An alternative to Spoike's suggested workflow (developer pulls from blessed repo, works locally, issues a pull request and finally the integrator pulls those changes & merges them) would be to use the continuous integration system as an intermediary.

As before, the developer pulls from blessed repo and works locally, but when done, they pull from the blessed repo again and merge themselves before pushing to an unblessed repo. Any changes to the unblessed repo are then reviewed (either manually or automatically) and moved to the blessed repo only if they are approved.

This means that the integrator only has accept or reject a change, not do the merge. ^{In my experience it is almost always better for the developer who wrote the code to perform the merge than for someone else to do it.}

As suggested in the mercurial book, hooks can be used to automate this procedure:

When someone pushes a changeset to the server that everyone pulls from, the server will test the changeset before it accepts it as permanent, and reject it if it fails to pass the test suite. If people only pull changes from this filtering server, it will serve to ensure that all changes that people pull have been automatically vetted.

Other issues

The problem of large test datasets can also be solved by putting that test data into a mercurial sub-repository. This will prevent the code repository getting bloated with test data, while still keeping the test data under revision control.

Answer 2

Okay, trying to answer this simply.

What you need to know

First thing you need to know: Mercurial is distributed version control and has some properties you should know about listed below.

• The source come from one repository, where that repository can be cloned. All cloned repositories can share code with each other through synching (with pull and push commands, that can be access restricted).
• Every user that has a copy of the code, has a clone of the repository. If they want to branch, they can do it in their local clone. That means you don't need to organize how every user should branch. They can do this for themselves.
• Tags are created in mercurial by a commit (which is the same as hard tags in git). This means you don't need a directory inside your repository structure for tags.

• The usual model that people work with in DVCS (which is employed in github and bitbucket) is to do it semi-centralized.

  Each user has a public repository (in some share or on a secure server) and a private repository (in their own workstations). They are both clones of an integrator's "blessed" repository. Whenever they feel they're ready to publish their code, they can push the changes from to their public repository. An integrator can then choose which users to pull code into the "blessed" repository.

  If the integrator can't merge some user's code easily, then the changes are rejected and it is up to that particular user to update their repository and fix the merge themselves. It is usually not that difficult if you merge often (as it is less code that needs to be merged) and usually that user should know what went wrong with the merge.

Repositories-per-project setup

So the usual set up is that for each project there is the following:

• A public read-only repository that the integrator is responsible for. It is "blessed".

  I.e. all users can pull/fetch content but have no access to push it.

• Each user can have their own public clone of the repository.

  Easiest set up as put in a share drive (though you might consider hosting such as bitbucket). The integrator receives pull-requests from the users and tries to pull the new code from these repositories. When merges are done without a hitch, it is put in the read-only repository. If not, then users are asked to fix the merge conflicts that arise by updating and merging it for themselves locally.

• Each user can have their own private clones of the repository.

  Good practice is to pull from their public clone, but it doesn't matter if they pull from their public or the integrator's. All commits are uniquely identifiable so merging commits that you forgot to fetch in the public one is relatively easy to fix (by pushing the changes from the private to the public, it automatically gets the integrator's changes as well).

Source code organization

As in how to arrange the project source itself is something you need to think through. If an artifact needs to be source controlled then put it in source control. Personally I don't like the idea of checking in artifacts that are made by the build or runtime (due to the high risk of merge conflicts on these kinds of artifacts) such as binaries or log files.

You can also check in configuration, as long as they make it easy for developers to get going and doesn't mess up configuration for releases or live/production environment (such as app/web server settings). This leads to the notion that if the configuration you have seriously hinders the developers to get started within five minutes after they've checked out the code then it needs to be refactored. Another requirements is that it should be darn hard for the developers to mess up the release or live/production environment.

You mention that you have test data that needs to be tied to some version of the code. Now this is a bit trickier because DVCS-systems such as Mercurial and Git have a tendency to get slow when you check in data that is HUGE. In my experience it gets really unbearable after 5 GB of binary files (your milage may vary, so you should test out how it works out for you). I would however recommend that you put generated data into it's own repository and have your test system tag them appropriately when checking them in (and/or create text files for the same meta data purposes).

I hope this all makes sense. Please comment below if I missed some detail or if something needs further explaining and I'll try to edit.