7

I have a medium sized Python program (12 KLOC) organized as a single Python package with multiple subpackages:

proj/
    setup.py
    proj/
        __init__.py
        projfile1.py
        subproj1/
            __init__.py
            subprojfile1.py
            ...
        subproj2/
            __init__.py
            ...
    tests/

I can see that some of these subpackages could be self-contained and could live in their own namespace:

proj/
    setup.py
    proj/
        __init__.py
        projfile1.py
        ...
    subproj1/
        __init__.py
        subprojfile1.py
        ...
    subproj2/
        __init__.py
        ...
    tests/

I'm thinking that breaking out subpackages might help reduce future complexity by reducing interdependence between separate parts of the program.

The only subproj1 imports outside of its namespace are utility functions from another subpackage in proj/

Are multiple toplevel packages appropriate for a Python project? Are there any projects that are organized in this way?

What are the drawbacks of organizing a project as multiple Python packages?

1
  • scikit-learn jumps to mind as a subpackaged project that appears sensibly constructed. I think the general answer to your general question is "it depends"; alas.
    – msw
    Nov 20, 2017 at 3:02

1 Answer 1

5

Multiple top-level packages are in principle OK, but I think they are rarely an appropriate solution.

By itself, a directory structure does not say anything about coupling or complexity. The difference between a directory structure x/a, x/b, x/c and a, b, c is just a directory. The design is still the same.

For an application, just do whatever is convenient for you – especially if this application is not distributed outside of your organization. If you find multiple top-level packages convenient, then use that project layout.

For code that will be used by other people, you have to be more considerate. The problem is that the directory layout corresponds to modules/packages inside your Python code. There is no way to resolve name clashes, so you should choose reasonably unique names for your modules. There can only be at most one top-level util module. Therefore, it is highly advisable that you namespace all your modules under your project's name. While Python/PyPI don't enforce that this module has the same name as the pip-installable package, this is a very sensible convention.

One detail where your directory structure does encourage coupling is the shared tests directory. It is often better to place the tests for each package within each directory. I.e., the tests for a don't go in tests/a but in a/test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.