8

Problem Statement

At our company, we have various application projects that we work on and then we also have libraries that those projects need to utilize. I feel the need (based on some similar question that have been asked) to say that none of these libraries are from a 3rd party, and that we design all of them in-house. Currently we do not control the version of these libraries whatsoever. We just plug all of the libraries that we need into the new application we are developing at the time, and just forget about it.

I don't think that this is the right approach. That being said, we do track the version of all of our application projects regularly through the use of Git. I am thinking that there must be a way to make sure that these stand-alone libraries are properly controlled.

Proposed Plan

What I am thinking to do is, have all of the libraries as there own repositories on our drive then just have a clone of each repository within the application working directory.

Question

Is my proposed plan okay? Or, should I not have revision control on these libraries in a stand-alone manner, then just check them into the application's repository? Something else?

  • 1
    This sounds like a case for git submodules or something like this: git-dependency-manager.info/use-cases/submodules – Jace Browning Apr 7 '16 at 17:55
  • @JaceBrowning does using Git submodules address the issue of checking out the proper libraries when it comes time to build (like 8bittree describes below)? – Snoop Apr 7 '16 at 18:06
  • Yes, submodules and other 3rd-party Git tools can check out particular versions of your internal libraries. – Jace Browning Apr 7 '16 at 18:11
  • @JaceBrowning I looked at the first link you posted, and as with that article being called "Replacing Submodules" I'd first like to familiarize myself with sub-modules before choosing an alternative. Do you think this is the right approach? – Snoop Apr 7 '16 at 18:14
  • Yeah, it's worth learning how submodules work. – Jace Browning Apr 7 '16 at 18:46
8

I would keep each library in its own repository. Start keeping track of library versions, for example with git tag.

A big problem with simply checking each library into each application's repository, is that you've essentially done copy and paste, and thus gain all the disadvantages that implies. Bugs fixed in the copy of the library in one application don't necessarily make it to the copies in other applications.

By having a single place for the library to officially live, you can more easily make sure all your applications can take advantage of bug fixes and new features.

For managing how each application gets a copy for actual build purposes, I'd recommend something similar to what the Cargo package manager does. Include a config file (xml, json, toml, etc) in each application's repository. In that config file, specify what libraries it needs, and what versions of those libraries it requires. You may also want to specify the locations of those libraries.

Then, either the developer or, preferably, a package manager, can read the file and clone the appropriate libraries, checkout the correct tag, and build them. You could clone them into a .gitignored subdirectory of the application. A dedicated library directory may be better, as then multiple applications that use the same version of the same library can share one copy.

  • Yes, this copying/pasting is what we've been doing all along and I think this is a serious problem. You revise the library, and all the other projects still have the old one (still containing the old bugs). – Snoop Apr 7 '16 at 18:03
  • 1
    The "preferably a package manager" part is the key. You need some sort of dependency management system, like NuGet for Visual Studio, or NPM for Node.js (or Bundler for Ruby... and the list goes on). – Greg Burghardt Apr 7 '16 at 20:12
  • @GregBurghardt I indeed use NuGet for package management, but if I'm not mistaken... All that does is go out and pull-in 3rd party libraries. So... can you explain what that has anything to do with the in-house libraries? – Snoop May 4 '16 at 11:42
  • @StevieV NuGet appears to support adding your own locations to fetch dependencies from. – 8bittree May 4 '16 at 12:24
  • 1
    @StevieV I probably wouldn't bother using both in one project, but if you're interested in a first hand comparison, it might be worth setting up one or two projects to use NuGet while leaving the others on submodules. With the second and later projects, you in theory won't have to go through the process of getting your libraries set up for NuGet, you just reuse what you did the first time. But overall, the difference between no package management and some package management is much greater than the difference between submodules and NuGet. – 8bittree May 4 '16 at 12:38
3

There are two basic approaches that you can take.

  1. Keep versions on libraries. Track dependencies carefully. You will eventually experience dependency hell.
  2. Keep all libraries up to date all the time. You will need very good unit tests, and a deployment strategy that provides a locked target for production.

I have worked at companies that use both approaches. For example Amazon tracks dependencies carefully, while Google keeps the world up to date.

My biased opinion is that the first approach pushes off your pain points, but in the long run it is more painful than the second approach. If you go with the second approach it is critical to automatically run unit tests, and I strongly recommend a good code review process. If you wish to copy Google, you can use https://www.gerritcodereview.com/about.md for code review and http://bazel.io/ for a build system. See http://googletesting.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html for a sense of how this scales to a large organization.

  • There are trade-offs for both. Versioned dependencies are great and allow you to innovate--right up to the point that you need to touch an unversioned resource or upgrade everyone quickly (new security requirements that require breaking changes, for instance). Unversioned libraries make development easy, but require lots of coordination and broad deployment strategies that are more than a little frightening. Having worked in both situations, I almost think a little of both is the best option. – mgw854 Apr 7 '16 at 20:45
  • @mgw854 Exactly. If you've got a micro-service environment, then each micro-service can go through its own QA/deployment cycle with a locked down set of libraries. This is a best of both worlds where development is easy, and production changes are more controlled. But then you have all of the complexity of managing micro-services. – btilly Apr 7 '16 at 21:19
  • I love versioned dependencies. I've never experienced dependency hell with in-house libraries, and I don't think it's a necessary consequence of that approach. I generally try to keep versioned in-house libraries up to date, though. – Marnen Laibow-Koser May 18 '18 at 0:06
  • @MarnenLaibow-Koser The problem comes with the fact that in-house libraries that have external dependencies. Once enough of them have an external dependency on something with a breaking change, you're then forced to choose between making your internal libraries incompatible, or sticking with the older version. It is really a choice between accepting pain now, or accepting more pain later. The decision can be put off for a surprising time..but not actually forever. – btilly May 30 '18 at 17:25
  • @btilly Of course that’s true, but that’s a consequence of depending on outdated external libraries. It has nothing to do with versioning your in-house libraries, which is what I thought we were talking about. – Marnen Laibow-Koser May 30 '18 at 20:19
2

I suggest that you develop your in-house support libraries with the same rigor that you would apply if you were to release those libraries publicly. That is, develop each library as a self-contained project with its own repository, issue tracker, documentation, packaging, build system, testing, versioning and release schedule. If you're the only user of the library, you have a little more freedom when changing things but high-quality stable APIs are still valuable.

Your application projects that consume those libraries can then pull them in just as any other third-party library. The mechanism by which that works will normally depend on the technology. For a library written in C or another natively compiled language, you would provide a tarball that can be unpacked, configured, built and installed. A small script that does this automatically will come in handy. Other technologies might have their own way of satisfying dependencies and build tools that might support fetching them automatically. (Like Maven does for Java.) In any case, you will have an FTP server or other internally shared resource where your build systems can fetch the dependencies they need. Whenever you feel ready to “release” a new version of a library, create a distribution archive of it and upload it to this server. (Under a unique name so the old versions are still available.)

Whether your applications always use the latest version of the library or switch less often can be decided on a per-application basis. It will probably be a good idea to “release” early and often and make applications adopt new versions as soon as possible but if your facing a timely deadline and a library had to introduce breaking changes, you might be happy not having to update right now.

By the way, did you consider making your in-house libraries public by releasing them as free software? It is generally reported that doing this helps improving the code quality and you might find other people who have the same needs and start collaborating with you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.