0

I have been debating this question internally for a few weeks now and I keep coming up short in finding a good answer to the question. I feel an example would be more descriptive than just talking about it so I will start with that.

I have a Python project repository structured as follows:

.
├── common_package
│   ├── __init__.py
│   └── other files and folders...
│
├── package_one
|   ├── src
│   |   ├── common_package -> ../common_package
│   │   └── package_one
│   │       ├── __init__.py
│   │       └── other files and folders...
│   ├── tests
│   │   └── ...
│   ├── README.md
│   ├── setup.cfg
│   └── setup.py
│
├── package_two
|   ├── src
│   |   ├── common_package -> ../common_package
│   │   └── package_two
│   │       ├── __init__.py
│   │       └── other files and folders...
│   ├── tests
│   │   └── ...
│   ├── README.md
│   ├── setup.cfg
│   └── setup.py
│
└── other packages with common_package...

Each one of package_one, package_two, etc... is a separate Python package with its own (wildly varying and fairly large) dependencies, build process, and distributions. However, all of these packages implement (in some form or another) things in common_package which provides a common interface to the different packages for competition in the future. The packages are completely disjoint other than that (barring a resources system managed by common_package).

Some packages are fairly small (few files each) so they do not warrant having a separate repository for each; also since their dependencies vary quite a bit and the techniques are very disjoint (and there is no use for any package in any other package), I do not think it would be wise to package them all under one setup.py umbrella.

I found this technique to be quite useful and practical, with one caveat: after installing a package, I now have common_package as an accessible, importable package from anywhere in my virtual environment. I fear this spells some bad news since the package name is very generic (in my implementation it is just common), and it could be overridden.

For example, this is how some code would look after I install package_one.

>>> from common_package import config
>>> from package_one import some_module
>>> some_module.do_stuff(config.SOME_PATH)

So my question is the following: is there any way I can improve on this structure? Particularly, can I someone encapsulate all of these sub-packages under one "namespace", such as project_name without having them be under one package that has to all be installed at the same time?

  • Is there a particular reason why the name of the project must be "common_package"? You aren't obliged to have a single project where everything in common is thrown in. Consider using multiple projects. – Neil Aug 30 '18 at 12:52
1

You can absolutely use namespace packages to group your packages under a project namespace. This also provides a solution for your common package: make it a dependency of your main packages. You would then have three packages, all with their individual setup.py:

  • project_name.package_one
  • project_name.package_two
  • project_name.common_package

It is perfectly fine to keep multiple Python packages in the same version control repository, though this might confuse some CI tools. You will also not be able to pip-install the version control repository directly. You might have to layer your own build system on top to verify that version numbers and other dependencies are in sync.

Note that namespace packages require you to represent the namespace package in your file system layout, e.g.:

common_package/
  setup.py
  src/
    project_name/
      __init__.py  # for namespace package
      common_package/
        __init__.py
        foo.py
        bar.py
  • This works very well for the most part, but I have a note: since Python 3.3 PEP420 introduced implicit namespace packages we do not need that __init__.py at all anymore. – Ziyad Edher Aug 30 '18 at 13:46
  • @ZiyadEdher Thank you for that link. The setuptool docs tend to be out of date, your best bet to do this is probably to look at large, actively maintained packages to see how they did this. – amon Aug 30 '18 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.