1

I'm designing a new language and the package-management system for it (something like NPM, Cargo, Pip, Gem, Cpan, Cabal, NuGet or the like).

I'm trying to decide what's a good way to handle the versioning of a package when only part of it is updated. I'm also curious to learn what's the best way to solve this problem with existing package managers.

Example of the problem

Let's suppose a package exports some types and functions that many other packages use:

# package: string 1.0

type String {
  length: Number,
  data: byte*
}

function from_NUL_terminated( data: byte[] ) -> String { ... }
function to_NUL_terminated( string: String ) -> byte[] { ... }
function from_repeated_char( length: Number, char: byte ) -> String { ... }
function get_UTF8_length( string: String ) -> Number { ... }
# ...

At some point, something in the "string" package needs to be changed. For instance a bug in from_NUL_terminated() gets fixed, or a new function get_UTF16_length() is added.

Because of this change, a new version of the "string" package is released: string 1.1.
This makes everything incompatible between the two versions: the language considers String from "string 1.0" a different type from String from "string 1.1". Even though the String type didn't change: only other stuff in that package did.

A consequence is that packages that depend on a different version of "string" cannot pass Strings to each other. A little function that is only used internally by a few packages causes a major split in the ecosystem of the language.

Obviously this problem must be avoided. How?

Further thoughts about the problem

From the example above it sounds like only types need to be protected from this issue. That's practical in most cases, but it's not technically correct. Ideally functions that don't change shouldn't be re-released in the new version either.

It sounds like every item in a package should have its own versioning, rather than package itself: every type, every function, every piece of data or metada.

What's the solution?

How do existing languages and their package managers handle this situation?

And how could I handle it for my new language and its new package manager? Can the package manager fix this problem on its own for any language, or would it need support from the language to do things properly?

Addendum: a minimal example showing the problem in Node.js

I created a project in Rust where two different versions of the crate "mystr" are used by the "main" crate and the "printerlib" crate: https://github.com/BlueNebulaDev/rust-version-test . (a crate is a package in Rust terminology)

The "mystr" crate exposes a type MyStr. This type is defined identically in both version. "mystr 2.0.0" also exposes a function that wasn't available in the previous version: mystr::from_slice().

The program can't compile, because the "main" package creates an mystr(1.0.0)::MyStr object and tries to pass it to a function that expects a mystr(2.0.0)::MyStr object. However MyStr never changes in the two versions: it's only other functions in the same crate that change.

I also created a Node.js project showing the same issue in that environment: https://github.com/BlueNebulaDev/node-version-test/ .

2
  • 1
    For versioning without declaring all version changes incompatible with each other, you may want to look at Semantic Versioning (semver.org) Jan 4, 2021 at 19:58
  • @BartvanIngenSchenau: I don't believe that it completely fixes the problem. Let's say that the function get_UTF8_length() needs to be removed from the string package. That's an "incompatible API change", thus it requires a MAJOR version. Nevertheless, String was not touched. Jan 4, 2021 at 20:26

1 Answer 1

3

the language considers String from "string 1.0" a different type from String from "string 1.1"

Well, you found a major flow in the language you're designing.

Types should be identified by their namespace. As soon as the namespace is kept unchanged between versions, a consumer would use string.String independently if the type is actually defined in version 1.0 or 6.5.0.814 of the library.

As for the package management systems, they should be smart enough when asked to install different versions of the same package. Usually, this is done by specifying not the hardcoded versions of the dependencies, but only the specific requirements. For instance, the dependency x may depend on package z version 3.0.2 or higher, whereas the dependency y may require package z version 3.4.*. The package management system understands then that it needs to install the version of z which is compatible with both requirements. Or, in other cases, requirements may be conflicting, and the package manager will halt and alert the user about the problem.

12
  • What makes you think that I know nothing about static linking and dynamic linking, other languages and package managers? I believe I already know about that. Yes, in most cases symbols inside an ELF or a PE are referred simply by their name (which might be a mangled variant of the actual identifier, depending on the language). But binary files don't contain information about types, they only contain data (e.g. strings), the compiled binary code of functions (text) etc. Type information, which is what I'm asking about is only stored in the package and doesn't have much to do with linking. Jan 4, 2021 at 20:12
  • 1
    “What makes you think that I know nothing about static linking and dynamic linking, other languages and package managers?”: oh, my apologies then. Reading your question, I had an impression that you weren't aware of all those things, because the sort of problems you describe in your question doesn't exist in many other languages. Jan 4, 2021 at 21:40
  • 1
    “something imported from base 1.0.0 is different from something imported from base 1.1.0 even though the "signature" is exactly identical”: obviously, since you're using two different packages, not two versions of the same package. What is possibly misleading is that Node is perfectly happy pretending that those two packages have the same name—it would be clearer if it would complain during the second require. Jan 4, 2021 at 21:40
  • 1
    Well, this is not a situation you'll find in production. Imagine that you import two packages, both depending on lodash. Either npm will figure out how to install one (and only one) version of lodash which makes happy both packages, or it will plainly tell that the requirements cannot be met. What you showed in your GitHub example is not a practical problem, but rather a theoretical thing where you tricked Node into calling two different packages by the same name. Jan 4, 2021 at 22:03
  • 1
    @BlueNebula Yes, and then the consumers of the library need to figure out what they'll do. It's a good thing if the package manager won't pretend it can ignore semver-major updates like they didn't exist. Jan 5, 2021 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.