Why do most package managers not keep a single reusable library with different versions of each package?

Question

When you use a Python virtual environment, you can activate it for different projects. Whatever packages are in it will be used by the project code. When you use npm, there will be a node_modules folder where the packages are kept, and this is per project.

In my mind, there's a really logical and straightforward way of handling packages: having a single library for all your projects, a list of requirements per project, and then the project uses the packages from that single library. I believe pnpm has this approach to some extent.

What I do not understand is: why is this approach not the norm? For instance, in Python, I don't even know of anything of the sort. Are there some obvious challenges with this or some obvious benefits to the other methods that I am missing?

Answer 1

Virtual environments (and equivalent) are supposed to be isolated. Sure, most package managers, even pip do download data into a single place. But then each virtual env gets its own isolated copy.

There are few benefits of this approach:

1. You can easily check what libraries are actually used by your project. It is almost always false that one keeps all dependencies in some list of requirements. That is technically rather hard, since libs depend on libs, which depend on libs, etc.
2. In fact, you can do the other way around: install dependencies first, and then generate requirements file out of it. This is very useful at the early stage of the project. It is also useful when you want to actually generate the true list of dependencies with fixed versions, so that every user works with exactly the same env.
3. Once you are done working on some project, you just delete it. And in case of Python additionally remove the virtual env. But how would you deal with those global dependencies? Are you going to manually go through each one of them and check if they are not used by other projects? Once I had such a mess on my pc that I didn't even know how many projects I have. So good luck with that. On the other hand: why would I keep around a library that I'm not using?
4. When you deploy your code, you really don't want some globally accessible cache on the server. Not when it hosts multiple projects. There's a security risk involved here.
5. Not to mention that dependencies are often specified by version range, not by concrete version. Say you deploy such code to the server. But server's global cache is different then yours locally. But both satisfy the range. You now may experience different application's behaviour, so error prone. This can (and should) be mitigated by things like docker, but docker is just a better isolated virtual environment, that can be easily moved around. That's all.

The only drawback is that it requires more disk space during development. And sometimes during deployment as well (assuming multiple projects live on the same server). But lets be honest: these are like what? Few wasted MB of disk space? Or maybe 100Mb? Or 1Gb? Noone really cares, disk space is the least expensive resource.