Recommended methodology for working with shared libraries and Mercurial

Question

I work in a small team of developers who all collaborate on several Zend PHP projects. We are using Mercurial with a collection of upstream repositories, as well as Jenkins for centralized testing and health reports. We want to implement a shared library of common classes, but are struggling with finding a workflow that works well with our current environment.

Currently, each project contains a structure like this:

- application        (project specific code)
- build              (temporary files generated by each build)
- library            (code used by multiple projects)
    - OurLibrary     (our company’s shared codebase)
    - Zend           (external libraries)
- tests              (test code specific to this project)

The library folder is excluded from testing, and so we have created a project specifically for running unit tests against the shared library. This decision was made primarily to avoid deploying the test classes by accident, something which can largely be mitigated by a modification to the automated build process, but also because we felt that it was preferable to keep the library lightweight.

We are including the library in each project as a Mercurial sub-repo. This means that changes pushed into the library from a developer working on ProjectA wil not automatically propagate to ProjectB (good) but will also not propagate to the library testing project (bad), and Jenkins will therefore not automatically warn us of potentially breaking code in the library (very bad).

An ideal situation would be that developers can commit changes to the library from any project, and that our automated tools would then run the latest library tests against the latest library code. Furthermore, this should not result in a version conflict when a developer pulls a new version of the library tests.

Is there an established methodology for this kind of workflow, or should we look into structuring our working practises a little differently before we fully commit ourselves to sub-repos?

Answer 1

If it were me I'd just set up the 'OurLibrary' project like any other. i.e. As a single project which includes tests.

I don't see how exposing a project's tests to other projects would ever be problematic: since tests are part of your documentation, I'd actually argue that it would be a good thing!

If that really isn't possible for some reason, another option to consider is to have projects use a compiled version of 'OurLibrary'. This has some drawbacks of its own though - e.g. having to manage the compiled version of the library; being unable to edit the 'OurLibrary' code project directly from other projects; etc.

Answer 2

The last resort

Consider that sub-repos are considered a feature of last resort.

This essentially means that you shouldn't use sub-repos unless you have a legitimate reason, you have a legitimate reason, so go for it.

When then?

I have a few rules of thumb for subrepos, so then I use them if and only if:

• I already have the subrepo as a stand alone repo (i.e. its not a library I may be incubating) AND
• (It is code I own OR the code in the sub-repo is being actively developed) AND
• Updating the code is not trivial AND
• (There is the potential situation where I may want to edit the subrepo code and push it back OR fork the subrepo code with a new project-specific while still being able to pull and integrate latest changes in that repo (keeps things flexible in emergencies and deadlines)).

Last reason...

Configuration management: when I want to track what versions of what coexist with which versions of whatever and —most importantly— I want to have control of these.

For example, say that you want to test your latest development/experimental branch of "OurLibrary" against your production code of your current project. Having subrepos in this situation is very useful.

Keep questioning yourself

You might get comfortable with the idea of using subrepos. Don't. Keep asking yourself "when should I not use subrepos?", if you don't come with a good reason just don't use them and everything will be fine.

Relax

Say that you didn't use subrepos and end up realizing you should've because reason X: you can always fix it later, in any case, you don't need the subrepo history for your current project.

In an extreme case where you wanted to remove the files where the subrepo was supposed to be since day 0, use the ConvertExtension to extract the subrepo with a filemap file.

I wouldn't recommend this for your case (where your main project is the parent of your desired subrepo) but it will be useful for the configuration management use case where you will have a new parent repo to handle versions between components and your main project is one of them (at the same level of the subrepo you want), e.g.:

project (repo)
|-component1
|-component2
|-component3

Turns into

project (new-repo)
|-component1 (subrepo)
|-component2 (subrepo)
|-component3 (subrepo)

Otherwise with:

project (repo)
|-app
|-lib
|--|-libA
|--|-libB

Turning into

project (repo)
|-app
|-lib
|--|-libA (subrepo)
|--|-libB

You will leave a hole in your history where libA doesn't exist up until you create the .hgsub entry.