How do I manage quickly changing python modules

Question

I write Python code for scientific computation. As it is research I face among other two problems:

1. the demands are quickly changing
2. results need to stay reproducable

Imagine you have a package A that is needed in an old project and then again in a new one. The new project however requires some changes (This would not be a problem if you could keep backwards compatibility. However, I experienced that this is practically impossible in small very specialized modules.). System wide installations would disallow to rerun the old code. So separate installations are required.

Currently, I solve this with git submodules: Package A is developed in a git repository and included as a submodule in both projects. This way, the code of the projects can always be run and updates of the package can be propagated easily.

However, I ran in a practical problem with python imports. The folder structure of a package A would look like this:

A - A - mod.py
 |   |- __init__.py 
 |- B
 |- C

Where B and C contain packages that are needed by A and the folders are added to the python path in init.py of A. However, this lead to a case where a package was used twice and with different versions:

A - A - mod.py
 |   |- __init__.py 
 |- B - B - mod2.py
 |   |   |- __init__.py
 |   |- C
 |- C

I.e. both A and B brought their "own" version of package C. When B adds C to the python path, the folder is ignored, because simply the folder is used, that was added by A. This can be fixed by specifying the correct path when importing. But it gets a bit ugly (see. https://stackoverflow.com/questions/34682638/is-there-a-convenient-way-to-translate-a-from-a-import-b-as-c-to-an-python-imp)

Due to this I started to ask myself if this is the right way to reduce code duplication while assuring a valid code base in a quickly changing environment. Do you see alternatives/better ways?

Answer 1

Here's an option: You can rename the new version of a package and alias back to the old name in modules where you wish to use the new functionality.

When Python upgraded the Unittest package in the standard library for Python 2.7, they made a backport available for Python 2.6 called unittest2 that was in the Python Package Index.

For Python 2.6 users, wanting to use the newer functionality in the newer module, it would be imported like this:

import unittest2 as unittest

and the new features from 2.7 were available in 2.6 as if you were using 2.7., and the old features continued to just work e.g.:

unittest.main()

You are like the Python 2.6 user who wants to use the newer version of the package.

When you upgrade a dependency, you can put a version number on the package name (as they did with unittest2), and then alias it upon import, until such time as you have all your code unittested and migrated to using the new module or you are confident enough to upgrade the core dependency, or (ugh) you could just do this indefinitely. I would hope you don't do this indefinitely, in my mind, it's just a stopgap.