Matching dependency versions across multiple Python projects

Question

I have about 60 repos containing Python packages, currently using setuptools in a setup.py (run via pip install) to manage third-party dependencies. Most of these packages need to be installed on a system at the same time, which means their dependencies all need to be compatible with eachother.

The dependency versions are specified with == because I want to lock the dependencies to specific versions, since I've tested the software against those specific versions, and occasionally a new version of a third-party package will introduce a bug.

I'm constantly struggling to keep dependent package versions in sync across all of the repos. For example, if I update somepackage from 1.22.1 to 1.22.2 in one project, I have to then go and modify the setup.py in a few dozen other repos to match.

My partial solution right now is to use a lot of virtual environments, but this isn't a complete solution because I still have to maintain compatible dependencies across all the packages within a given venv.

Also, I'm really trying to move away from having separate venvs for various reasons, including:

• It's inconvenient to have to be constantly switching venvs.
• The entry point scripts that get installed are in the venv's bin, but I'd rather have all applications be installed globally. I can write scripts that switch to the venv then run the application and install those scripts in a global location, but that makes the deployment workflow very clunky.
• If I release an update to one of my packages, I have to install the update in every venv, instead of just once globally. This is inconvenient, error-prone, and time consuming.

My question is: Is there some way to more conveniently manage third-party dependency versions across multiple repos that share some of the same dependencies? Using the same example, if I update somepackage from 1.22.1 to 1.22.2, I'd really like to just change that version number in one place. If it matters, I'd like to support Python back to at least 3.9.

The only possible solution I've been able to come up with is something like:

• Have a package, let's call it mysetup, that exports a bunch of constants containing the version numbers of any dependency that any of the repos use.
• Import that package in setup.py and use it to construct the install_requires argument to setup.

But the main problem with that is that mysetup then needs to be manually installed first before installing any of the repo packages, which kind of breaks the deployment flow a bit.

Is there any solution to this?

Answer 1

60 repos containing Python packages

I have a few years' experience working with about double that. Different repos had different policies. Many of them would rev independently. Keeping a repo small and focused leads to fewer deps and thus fewer conflicts.

dependency versions are specified with ==

I support that "strict" policy for a shipping product, especially when the version numbers are copied out of a running test environment. It tends to be inconveniently restrictive in a development environment. Admit point-release bugfixes at the very least, and consider allowing the minor version to be bumped, as well. Let's describe either of those as a "loose" approach. And using e.g. >= 1.2.3 would be a "liberal" one.

trying to move away from having separate venvs

I am skeptical that you will ever get down to just one. Changing your deps implies running 60 test suites to approve that change. Consider embracing N separate venvs, even if N is less than 60.

Minimally you will need to support the last 2 cPython releases, implying a minimum of 2 separate venvs, an "old" and a "new" one.

In any event, you will certainly need an automated way to

1. push a dep version change into multiple repos,
2. test in each repo, and
3. accept / rollback the change.

You mentioned local package mysetup. I encourage you to use one or more packages like that in a different way. In mysetup's setup.py have it depend on specific (==) install_requires versions. Then have your other repos depend on mysetup, so the {pip, conda, poet} solvers will transitively pull in many deps. You could even publish the (very thin!) mysetup package on pypi if you like.

In practice I believe you'll want more than one such package, perhaps with one focused on core functionality including requests, another that pulls in {sqlalchemy, mysql, postgres}, and another that pulls in things like {numpy, polars, pandas}. Smaller chunks leads to less DLL hell, fewer conflicting requirements.

---

updating

Every now and again turn all the strict == deps into liberal >= ones, and take advantage of pip's solver.

Verify your tests succeed.

Commit the new version numbers as strict == dependencies. For example pip freeze can produce such a list, which will then be easy to git diff against the previous set of known-working versions. Updating frequently means small, low-risk changes. Conversely, waiting months or years between updates invites all manner of complex incompatible version constraints to creep in, which you get to diagnose and deal with.

---

simultaneous update

If you have a couple dozen deps, it's pretty important that several times per year you update all of them simultaneously to most recent release, and test that it works.

I have seen too many repos attempt the following approach: Add strict == dep A, then weeks elapse. Later add strict dep B, later C and so on. Notice that C needs another version of A, so upgrade both. Eventually packages B and Z are years out of date, and the solver is faced with hopelessly incompatible constraints. You could try a liberal approach at that point, risking that many packages have issued major releases incompatible with your app. Nightmare ensues.

Keep your app reasonably up-to-date w.r.t. its deps. And don't let A, B, ... Z drift months or years apart from one another's release dates. They each have their own graph of dependencies. You want to test against version combos that the upstream authors also tested against. Holding back one library to be years before another library's release means you get to test that unique combination, and diagnose issues.

---

reports

Consider writing a script that does sort | uniq on a bunch of requirements.txt files, producing a single "superset" of requirements. If pip finds a solution, propagate the result back into strict == requirements in each repo. If not, then we probably want smaller chunks, increasing our chances of finding a valid solution.

Also, when your CI/CD pipeline successfully finishes running an automated test suite, take care to record the strict version requirements of deps, in order to bake them into your release process.