Is checking dependencies into source control worth the cost?

Question

For the last several years, I've been checking all dependencies of my team's nodejs project into source control. At first, we stored the archives of all dependencies using yarn v1's offline mirror, now we store zips using yarn v2's cache feature.

The reason I felt like it was a good idea to do this is package security, namely:

1. We can run CI builds completely without internet connection using vetted packages.
2. We're insulated against package loss: if someone were to unpublish their work, or there would be an npm registry outage, we'd still have access to packages to perform builds.

These feel like good reasons; however, what I didn't realize until well after this decision were the costs of the decision:

• Network speed in many scenarios is faster than disk speed: it's often faster to fetch all packages from a CDN than to decompress them on disk and restructure into node_modules
• Storing archives inside git balloons repository size significantly, reducing CI speed non-trivially through slower clone times (even shallow clones are slow because of the size of the dependency tree, though this may be limited exclusively to the train-wreak of the JavaScript ecosystem).

(1) is covered generally by a lock file: storing hashes of repositories to avoid malicious attackers changing packages. (2) is a risk but perhaps a fairly low one all-things-considered.

I think this question almost delves into the range of opinion, but I hope just slightly enough to be an OK question here, which is: is it more beneficial to check project dependencies into source control or to fetch them with a proper lock file? An extension of this is: if you do check dependencies into source control, what are the best practices to maximize benefits and minimize costs?

Answer 1

It seems there are two problems:

1. Tracking NPM dependencies in version control is clunky and slow.
2. Unzipping all of these dependencies is a bit slow.

These are all solvable problems without a major overhaul to your development ecosystem. Any dependency management solution with a public internet component suffers from these same problems, and they all basically have the same solution. This affects any tech stack with a dependency management tool like NPM, so you can apply this basic information to NuGet, RubyGems, PHP Composer and many others.

We can run CI builds completely without internet connection using vetted packages.

This is neither a good nor bad reason. It depends on the sensitivity of your codebase and how security-minded you need to be. Most applications do not require a build server cut off from the public internet. Hosting your own package server inside your network can eliminate the need for the public internet anyhow. This allows you to create private NPM packages, including known good copies of 3rd party packages originally downloaded from the public NPM registry.

We're insulated against package loss: if someone were to unpublish their work, or there would be an npm registry outage, we'd still have access to packages to perform builds.

With a private NPM registry you will be insulated from a public NPM registry outage. You can keep your build server cut off from the internet and guard against packages disappearing.

Network speed in many scenarios is faster than disk speed: it's often faster to fetch all packages from a CDN than to decompress them on disk and restructure into node_modules

Disk speed is an order of magnitude faster than network speed with today's technology. The slowdown is probably caused by the time it takes to decompress the zip files and write the contents to disk. As with all performance problems, measure the difference. If the difference is measured in seconds I wouldn't even worry about this. You've got bigger problems to solve with a higher payoff.

Storing archives inside git balloons repository size significantly, reducing CI speed non-trivially through slower clone times (even shallow clones are slow because of the size of the dependency tree, though this may be limited exclusively to the train-wreak of the JavaScript ecosystem).

This is something I have to agree with. This is also something a private NPM registry can solve, since NPM packages can be cached locally on the build server, or downloaded over a (presumably) local network.

Answer 2

What to Source Control

In an ideal world it would be great to have a single source of truth within which all the information necessary for making a program exists.

But that isn't quite right. We don't want to check the entire OS, Drivers, and even the specific hardware into source control. That last case isn't even possible at the moment till someone comes up with a perfect atomic scanner.

So what is it that we actually want?

I think it boils down to fragility.

Consider a Reef versus the Open Ocean.

A reef provides many services to the fish that live in it. Each species can highly specialise, and not have to worry to much about some concerns.

On the other hand a fish in open water has to solve many more problems: Shelter (or lack of), Food source (finding it), Defence (you better be able to flee or fight). Although some problems are still taken care of (gravity).

EG: clownfish don't have to worry too much about surviving anything other than the anemone they live in. Problems arise though when they aren't in a anemone, even if they are in a reef.

Taking this back to the code:

1. There exists one or more environments in which the application is expected to execute. Lets call these the production platforms. EG: .NET, Java, Win OS, Mac OS, *nix, ..., LISP machine.
2. There exists one or more environments in which the application is to be built, and packaged. Lets call these the build platforms.

These are the reefs in which your program can be assembled and be operated.

Now which reef are you in, and which reef do you want to be in?

If you are comfortable with operating in only highly specialised platforms, then it makes sense to leverage all of those platform services.

• On the plus side, many of those services are free, or very inexpensive to use.
• On the con side, if global warming comes along and kills your reef... Sorry if technology moves on and your platform dies then you have issues. The application is adapted to that environment.
  • If the application was engineered to be adaptable, then with some pain it can transition to a different reef, with slightly different services.
  • If the application is bank rolled, then it might be possible to subsume services in the old environment into your code, enough to essentially recreate a mock ecosystem around itself elsewhere.
  • Otherwise the application will die.

If you are comfortable with an anaemic platform, then you are going to have to work a lot harder.

• On the plus side, shocks are much less likely. Fewer dependencies, means a smaller area in which change can hurt you.
• On the con side, a lot more stuff becomes your code. That codes needs maintenance.

---

Personally

I subscribe to the early Excel teams perspective of source control. Everything above the OS and the vanilla C compiler is their code.

• Source Code
• Data
• Configuration
• Platform Agnostic Code
• Platform Specific Code
• Documentation
• Build System

What did it buy them?

• The world's most prolific Functional Programming Environment
• Entrenchment within the Business Community as a reliable spreadsheet
• The ability to port identical UI, and functionality across Operating Systems and Versions of those systems.

---

NPM/Yarn

Store the unzipped source/libraries not a zip file.

Most files between releases do not actually change. Unzipped, git will detect and ignore these.

If speed is an issue, I'd look at the root cause. Namely the megabytes of dependencies in NPM/Yarn. Are they needed? Can they be reduced? Are you relying on a entire library for 5 lines of code?

Answer 3

One way to do this is that you have a someone responsible for providing compiled libraries with sets of headerfiles, and that person checks the compiled libraries into git. It’s up to them to organise the sources. So this all doesn’t interfere with your main work.