What is a proper way to structure a Python project consisting of smaller packages that have shared code?

Question

I have been debating this question internally for a few weeks now and I keep coming up short in finding a good answer to the question. I feel an example would be more descriptive than just talking about it so I will start with that.

I have a Python project repository structured as follows:

.
├── common_package
│   ├── __init__.py
│   └── other files and folders...
│
├── package_one
|   ├── src
│   |   ├── common_package -> ../common_package
│   │   └── package_one
│   │       ├── __init__.py
│   │       └── other files and folders...
│   ├── tests
│   │   └── ...
│   ├── README.md
│   ├── setup.cfg
│   └── setup.py
│
├── package_two
|   ├── src
│   |   ├── common_package -> ../common_package
│   │   └── package_two
│   │       ├── __init__.py
│   │       └── other files and folders...
│   ├── tests
│   │   └── ...
│   ├── README.md
│   ├── setup.cfg
│   └── setup.py
│
└── other packages with common_package...

Each one of package_one, package_two, etc... is a separate Python package with its own (wildly varying and fairly large) dependencies, build process, and distributions. However, all of these packages implement (in some form or another) things in common_package which provides a common interface to the different packages for competition in the future. The packages are completely disjoint other than that (barring a resources system managed by common_package).

Some packages are fairly small (few files each) so they do not warrant having a separate repository for each; also since their dependencies vary quite a bit and the techniques are very disjoint (and there is no use for any package in any other package), I do not think it would be wise to package them all under one setup.py umbrella.

I found this technique to be quite useful and practical, with one caveat: after installing a package, I now have common_package as an accessible, importable package from anywhere in my virtual environment. I fear this spells some bad news since the package name is very generic (in my implementation it is just common), and it could be overridden.

For example, this is how some code would look after I install package_one.

>>> from common_package import config
>>> from package_one import some_module
>>> some_module.do_stuff(config.SOME_PATH)

So my question is the following: is there any way I can improve on this structure? Particularly, can I someone encapsulate all of these sub-packages under one "namespace", such as project_name without having them be under one package that has to all be installed at the same time?

Answer 1

You can absolutely use namespace packages to group your packages under a project namespace. This also provides a solution for your common package: make it a dependency of your main packages. You would then have three packages, all with their individual setup.py:

• project_name.package_one
• project_name.package_two
• project_name.common_package

It is perfectly fine to keep multiple Python packages in the same version control repository, though this might confuse some CI tools. You will also not be able to pip-install the version control repository directly. You might have to layer your own build system on top to verify that version numbers and other dependencies are in sync.

Note that namespace packages require you to represent the namespace package in your file system layout, e.g.:

common_package/
  setup.py
  src/
    project_name/
      __init__.py  # for namespace package
      common_package/
        __init__.py
        foo.py
        bar.py