I write Python code for scientific computation. As it is research I face among other two problems:
- the demands are quickly changing
- results need to stay reproducable
Imagine you have a package A that is needed in an old project and then again in a new one. The new project however requires some changes (This would not be a problem if you could keep backwards compatibility. However, I experienced that this is practically impossible in small very specialized modules.). System wide installations would disallow to rerun the old code. So separate installations are required.
Currently, I solve this with git submodules: Package A is developed in a git repository and included as a submodule in both projects. This way, the code of the projects can always be run and updates of the package can be propagated easily.
However, I ran in a practical problem with python imports. The folder structure of a package A would look like this:
A - A - mod.py | |- __init__.py |- B |- C
Where B and C contain packages that are needed by A and the folders are added to the python path in init.py of A. However, this lead to a case where a package was used twice and with different versions:
A - A - mod.py | |- __init__.py |- B - B - mod2.py | | |- __init__.py | |- C |- C
I.e. both A and B brought their "own" version of package C. When B adds C to the python path, the folder is ignored, because simply the folder is used, that was added by A. This can be fixed by specifying the correct path when importing. But it gets a bit ugly (see. https://stackoverflow.com/questions/34682638/is-there-a-convenient-way-to-translate-a-from-a-import-b-as-c-to-an-python-imp)
Due to this I started to ask myself if this is the right way to reduce code duplication while assuring a valid code base in a quickly changing environment. Do you see alternatives/better ways?