Version control on standalone (in-house) libraries?

Question

Problem Statement

At our company, we have various application projects that we work on and then we also have libraries that those projects need to utilize. I feel the need (based on some similar question that have been asked) to say that none of these libraries are from a 3rd party, and that we design all of them in-house. Currently we do not control the version of these libraries whatsoever. We just plug all of the libraries that we need into the new application we are developing at the time, and just forget about it.

I don't think that this is the right approach. That being said, we do track the version of all of our application projects regularly through the use of Git. I am thinking that there must be a way to make sure that these stand-alone libraries are properly controlled.

Proposed Plan

What I am thinking to do is, have all of the libraries as there own repositories on our drive then just have a clone of each repository within the application working directory.

Question

Is my proposed plan okay? Or, should I not have revision control on these libraries in a stand-alone manner, then just check them into the application's repository? Something else?

Answer 1

I would keep each library in its own repository. Start keeping track of library versions, for example with git tag.

A big problem with simply checking each library into each application's repository, is that you've essentially done copy and paste, and thus gain all the disadvantages that implies. Bugs fixed in the copy of the library in one application don't necessarily make it to the copies in other applications.

By having a single place for the library to officially live, you can more easily make sure all your applications can take advantage of bug fixes and new features.

For managing how each application gets a copy for actual build purposes, I'd recommend something similar to what the Cargo package manager does. Include a config file (xml, json, toml, etc) in each application's repository. In that config file, specify what libraries it needs, and what versions of those libraries it requires. You may also want to specify the locations of those libraries.

Then, either the developer or, preferably, a package manager, can read the file and clone the appropriate libraries, checkout the correct tag, and build them. You could clone them into a .gitignored subdirectory of the application. A dedicated library directory may be better, as then multiple applications that use the same version of the same library can share one copy.

Answer 2

There are two basic approaches that you can take.

1. Keep versions on libraries. Track dependencies carefully. You will eventually experience dependency hell.
2. Keep all libraries up to date all the time. You will need very good unit tests, and a deployment strategy that provides a locked target for production.

I have worked at companies that use both approaches. For example Amazon tracks dependencies carefully, while Google keeps the world up to date.

My biased opinion is that the first approach pushes off your pain points, but in the long run it is more painful than the second approach. If you go with the second approach it is critical to automatically run unit tests, and I strongly recommend a good code review process. If you wish to copy Google, you can use https://www.gerritcodereview.com/about.md for code review and http://bazel.io/ for a build system. See http://googletesting.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html for a sense of how this scales to a large organization.

Answer 3

I suggest that you develop your in-house support libraries with the same rigor that you would apply if you were to release those libraries publicly. That is, develop each library as a self-contained project with its own repository, issue tracker, documentation, packaging, build system, testing, versioning and release schedule. If you're the only user of the library, you have a little more freedom when changing things but high-quality stable APIs are still valuable.

Your application projects that consume those libraries can then pull them in just as any other third-party library. The mechanism by which that works will normally depend on the technology. For a library written in C or another natively compiled language, you would provide a tarball that can be unpacked, configured, built and installed. A small script that does this automatically will come in handy. Other technologies might have their own way of satisfying dependencies and build tools that might support fetching them automatically. (Like Maven does for Java.) In any case, you will have an FTP server or other internally shared resource where your build systems can fetch the dependencies they need. Whenever you feel ready to “release” a new version of a library, create a distribution archive of it and upload it to this server. (Under a unique name so the old versions are still available.)

Whether your applications always use the latest version of the library or switch less often can be decided on a per-application basis. It will probably be a good idea to “release” early and often and make applications adopt new versions as soon as possible but if your facing a timely deadline and a library had to introduce breaking changes, you might be happy not having to update right now.

By the way, did you consider making your in-house libraries public by releasing them as free software? It is generally reported that doing this helps improving the code quality and you might find other people who have the same needs and start collaborating with you.