Branching and versioning strategy for shared libraries

Question

These posts seem related, but my brain is starting to melt, trying to think this through :P

My employer has just started using source control, primarily because before they hired more developers, the "repository" was the hard drive of the lone dev, who works mainly from home. All of the .NET code he'd written was checked in en masse, and there is a lot of duplicated (read: copy-pasted) functionality. Right now, our SCM system is glorified backup.

I'd like to pull some of the duplicated code into shared libraries. I'm leaving the original repo alone so that we don't break anything—we can move and/or refactor existing code as and when needed. So I've set up a repo just for new code, including libraries.

My problem revolves around versioning the libraries without miring us in excessive process: having acknowledged that we need a more coherent approach, with more devs all writing very similar code, management and the other devs are open to reorganising things, but it probably won't go down well if the solution starts to affect productivity.

The ideal solution in my obsessive mind is to build the libraries separately, with each dependent project built against deliberately chosen, compatible versions of them. That way, we know exactly which clients have which versions of which libraries, can more reliably reproduce bugs, maintain independent release branches for products and libraries, and not break each other's projects when changing shared code.

This makes updating the libraries cumbersome though, particularly for the dev who works from home. I expect the libraries to change quickly, at least initially as we (finally) pull the common bits together. It's quite possible that I'm completely overthinking this and we'd be fine just building everything against the most recent library commits, but I'd like to at least be prepared for the day we decide that some components have to be independently versioned and distributed. The fact that some of the libraries will have to be installed in the GAC makes versioning particularly important.

So my question is: what am I missing? I feel like I've fixated on one solution and am now trying to find variations on it that will make changes smoother. What kinds of strategies have you used to solve this kind of problem before? I realise this question is all over the place; I'll do my best to clean up and clarify any points of uncertainty.

And as much as I'd love to use Mercurial, we've already spent money on a centralised commercial SCM (Vault) and switching isn't an option. Besides, I think the issue here goes deeper than choice of version control tool.

Answer 1

I commend your initiative. This should yield a number of benefits for the organization as you implement this. I would move the code rather than copy it. Having copies will likely result in incompatible changes which will need to be resolved.

There will be some pain as the libraries are developed and stabilized. Once that is done, the benefits will arrive. Remember that the interfaces to the library are essentially a contract with the projects working with the contract. You may have the advantage of removing old interfaces as you may be able to determine if they are used.

While the libraries are stabilizing, getting new libraries should probably be part of the code update process. You may want to schedule commits of library changes. Announcing the code being moved may help reduce conflicting changes.

The libraries should be treated as separate projects and stabilized as soon as possible. Once they are stabilized (particularly the interfaces) it should be easier to integrate changes with other projects. New libraries should work with old code. Tag stable library releases with their own release id. Try to treat the libraries as you would third party libraries.

Answer 2

Code that is shared between projects must be treated as projects on themselves. They must be treated with the same thoroughness as third-party libraries. There's no other way.

If you can't get your group to adopt a strategy for shared code, you can adopt one yourself with the help of modern source code management tools like Mercurial or GIT, even if neither is the SCM your company will officially use. With a little care, two SCMs can use the same work directory just by telling one to ignore the internal files of the other. You'd use one SCM to deal with the everyday, and the company one to integrate.

At any rate, you must be in charge of when to update your working directory with the modifications other projects have made to the shared code. Those should happen only when you're ready to deal with the incompatibilities that may arise; otherwise, your work may become unstable beyond doable.

Answer 3

If you have the ability to break them up into separate "module" projects, I would take that approach. One thing to be concerned about is code coupling. You would create a separation of concerns by breaking them up, which is a good practice.

Some benefits:

1. Simpler debugging of each module.
2. Faster build cycles (no rebuilding of libraries that have not changed)
3. Once something works you're less likely to break it by changing another area outside of that module (not 100%, but you reduce the chances).
4. Easier to break up work assignments if it's not a large interdependent mess.
5. Faster distribution of smaller pieces when you fix one library (a big reason why you have .dll/.so files).

I do have one question about this part:

"The ideal solution in my obsessive mind is to build the libraries separately, with each dependent project built against deliberately chosen, compatible versions of them."

Do you static link the whole project? If so this would lead to: 6. Reduction of unnecessary code bloat.

Some Negatives:

1. If the project is small, then you're just adding complexity where it is not needed.
2. Branching lots of modules can turn into a maintenance problem for really large projects. I don't know "Vault", so I'm not sure if their branching operations are good or bad.

Answer 4

Right now, it looks like you have one codebase is built all at once into one target. You probably have no more than five developers. I don't really see the utility of separating the libraries too much. You'd change your workflow from code -> compile -> run to code -> compile -> copy DLL -> compile -> run...ick.

Some companies (Google and Amazon) have sufficient infrastructure for development that it's pretty painless to have lots of separate libraries built separately. The infrastructure to make it painless involves a way to specify library versions that lives in your SCM (which you probably have), a way to specify dependency versions that lives in your SCM, a build server that can make sense of it and compile everything properly, and a way to grab the appropriate build artifacts from your build server and from your local workspace. I doubt you have that.

Without that infrastructure, I'd separate the project when one of the following applies:

• You have multiple products or distinct applications depending on this library
• You work exclusively on these libraries for several days at a time
• The library's functionality is extremely separate from anything business-related (for example, wrappers around platform-specific APIs)
• Build times for the combined project grow too large

I'd worry about building that infrastructure when you have a lot of projects and some have dedicated developers and independent release cycles.

What you can and should do now is set up a build server for reliable, repeatable builds and make sure that you can find the source revision from a given executable. You seem to be using .NET; it's pretty straightforward to set up CruiseControl.NET to insert a version string, including the revision number.

Once you have a build server, splitting off a library will pretty much be a matter of moving it in Vault, adding another target in CC.NET, and copying the resulting DLL into your main project's library folder and committing it. Easier on the SCM side than day-to-day development.

Answer 5

Mercurial has a feature called subrepositories. I recently read this blog from Kiln explaining how they work.

Basically, you link your project to a specific revision of a library repository. You can update the library as much as you want without breaking the dependent projects. When you are ready, you can pull in the new features of the library into your project and deal with any breakage. Different projects can link to different revisions of the library, so they don't all have to be in sync and use the same library revision.

Maybe Vault has a similar feature.

Answer 6

We've added a metadata file to each module in SC that indicates the name of all the other modules it depends on. A product will have a second metadata file indicating which version of each dependent module is required for the product. On top of this is a tool to analyse the metadata files, check out all of the required modules, and generates a project for our IDE.

This ensures our builds are always reproducible, and that the correct header files are always shared between the components. Other complexity can be layered on as the need develops. We have tools that can now generate reports on the differences between any two builds of a product, specifying source files that changed, check-in comments, version comments, authors, across all of the independent modules in SC. We get a phenomenal amount reuse from our code base.