1

I know with NPM caret, tilde and some logical operators can be used to specify version ranges. This post explains a bit on how this works.

The problem now is I find it hard to reconcile the use of version ranges with the idea of having reproducible builds. Version ranges for dependencies means that you are not specifying a requirement for a particular version but a range of versions, which might change between builds (e.g. a patch release of a dependency was released between the last and current build).

Reproducible builds seek to remove variability in the environment ensuring that every repeated build would always be the same.

How it is ever possible to have reproducible builds with version ranges when using NPM?

  • 1
    With the yarn.lock or package-lock.json, which contains the explicit versions resolved. – jonrsharpe Aug 13 '20 at 15:23
  • but the yarn.lock or package-lock.json are created from information in package.json what is stopping them from also changing on the build server? – Finlay Weber Aug 13 '20 at 15:38
  • 1
    What's preventing anything else in your version-controlled code from changing on the build server? Hopefully you're just not doing that. Running npm ci for example will not update the lockfile and will bail if it doesn't match the package file. – jonrsharpe Aug 13 '20 at 15:41
  • They are created by the dependency resolver. Just don't run the dependency resolver. Note that if your build process is changing files inside of the repository, you have much bigger problems than worrying about version ranges. – Jörg W Mittag Aug 13 '20 at 15:41
  • 1
    npm install is fine if the package-lock.json is compatible package.json. See e.g. stackoverflow.com/a/53594050/3001761. Generally prefer npm ci in CI - it's safer and faster (and is the default in Travis if it's usable). – jonrsharpe Aug 13 '20 at 16:06
4

It's possible by having two distinct lists of dependencies, one with ranges, and one with specific versions, known as a lockfile.

Version ranges are helpful for libraries so that bug patches in your libraries' dependencies can be included without the libraries needing to be updated. They also give you the option of choosing a library version that meets the requirements of multiple packages.

Reproducible builds are generally more useful for apps than libraries. So by committing the lockfile to source control and including it in your deployment system, you can ensure that the precise versions of all dependencies you tested locally will be used when deploying, whether to one server or a thousand.

Npm keeps the lockfile as a separate file, so that you can choose whether to use it or not. If you always want the latest versions, you can have them. If you want reproducible builds, you can have them by distributing the lockfile.

  • But the processes of building on the build server cant lead to downloading a newer version of a dependency, which sure will also update the lockfile, hence such a build is no longer identical to the build at the point where the developer checked in the code. Right? or am I looking at this the wrong way? – Finlay Weber Aug 13 '20 at 15:37
  • 4
    Only if you explicitly tell it to re-resolve the dependencies. But then, you explicitly asked for it. Note that if your build process is changing files inside of the repository, you have much bigger problems than worrying about version ranges. – Jörg W Mittag Aug 13 '20 at 15:40
2

You can have a reproducible build and version ranges for your dependencies, so long as each build does not reinstall dependencies.

The version range is a setting in a config file, but a build is more than a config file. If the code is exactly the same, including dependencies from NPM, then it is a reproducible build. The challenge is to only download those dependencies once, and reuse them for each build. As soon as you remove those files and the build does another npm install you are not guaranteed to have a perfectly reproducible build.

Version ranges are a convenience. Some projects do not need a perfectly reproducible build. Others do.

Personally I would be surprised to find version ranges for dependencies when a reproducible build is desired. I would expect exact version numbers, and any change to those versions be planned as part of the development process, which sets a new baseline for reproducible builds. By specifying a version range, you are relying on build logic alone to ensure reproducibility.

Even if you specify exact version numbers, the mere fact an npm install downloads the source files again is a bit of a red flag. Perfectly reproducible builds should not contain any code changes, and may also involve calculating a checksum of the files used to produce the build.

NPM is not something you control, so there is nothing truly preventing someone from deleting their package and then republishing it under the same name, same version, with different files. So even exact version numbers for dependencies are not fail-safe.

It depends on how you define "reproducible." With a narrow version range, you might only receive patches, but no new features. This is not exactly a terrible thing, nor is it particularly great. It depends on the project.

If "reproducible" means bit-for-bit any two builds are exactly the same, I would argue exact version numbers for dependencies are not enough, but a good start. You may also need to calculate a checksum based on the files used to build the application.

  • 1
    "republishing it under the same name, same version, with different files" - the lockfile includes a hash, so although you wouldn't be able to repeat the build if a version was re-released with new files and you had no local copy, you would know about it. Likely if you're this interested in reproducibility you're not relying solely on the public registry, though (or necessarily on being able to rebuild, for that matter, rather than storing the result in an artifact repo). – jonrsharpe Aug 13 '20 at 15:26
  • @jonrsharpe: yes, that's a good point. Don't download stuff from the public internet if you want that level of accuracy. A private, in-company NPM repository would be a better way to go. – Greg Burghardt Aug 13 '20 at 15:34
2

The problem now is I find it hard to rectify the use of version ranges with the idea of having reproducible builds.

It is obviously impossible.

how it is ever possible to have reproducible builds with version ranges when using npm

You can't. What makes you think you can?

This doesn't even have anything to do with NPM. It is just basic logic. "Reproducible build" means all versions are exactly specified, "version range" means, at least one version is not exactly specified, ergo you cannot have reproducible builds with version ranges, and you cannot have version ranges with reproducible builds.

That's just a simple logical contradiction. Nothing to do with NPM or any other packaging system.

If you want reproducible builds, you need to give up on using version ranges. Nobody is forcing you to use version ranges, and nobody is forcing you to use reproducible builds. You can choose to use either one of them, or neither one of them. You can not choose to use both, because they logically contradict each other.

What you can do, however, is the following:

Most newer package management systems split dependency resolution and package build into two separate steps. RubyGems, NPM, Yarn, and others do this, for example.

In NPM, RubyGems, and Yarn, after the package management system has resolved the dependencies, it records the resolution it found. All three package management systems I mentioned call this a "lock file". The package build is then performed using the already resolved dependencies from the lock file.

So, you can use version ranges during dependency resolution, but then you can commit the lock file to the repository, and thus everybody who builds the package from the repository, will get the already resolved exact versions.

Again, that is your choice. You can distribute the lock file, but you don't have to.

Again, all of this is a choice. Some people like version ranges, some people don't. If you don't like 'em, don't use 'em. Some people like reproducible builds, some people don't. If you decide to have reproducible builds, then you can't have version ranges. Period.

I, personally, chose to re-resolve dependencies on every build for my own private packages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.