How to structure Python modules so they are accesible from different submodules

Question

This is somewhat similar to this question but I could not find a solution there.

I have a project that I've worked on over the past 4 years. I started without any Python knowledge and learned as I went along, so there is a lot of legacy code and weird things that work but could break easily. I'm redoing this project with my current knowledge hoping to get a better system.

This is an embedded project that runs on a small linux board. I have my main "core" system (with its repo) and several subfolders:

• main folder: queue script, worker script
• p_connections (database, sftp and ssh libs)
• p_hardware (gpio access, adc, other busses)
• p_system (file system etc)
• p_worker (measurement scripts)
• p_logging (log handlers)
• p_sensors (hw access to the sensors and processing for their raw data)

Some of these modules are used elsewhere (in other projects) as well (such as p_sensors) and I plan to incorporate them as a GIT submodule (as that seems to be the correct way to do this?).

However, some of these submodules need access to other submodules.

Some examples:

• The credentials modules needs to be able to read the database (p_connections module) but also log (p_logging module).
• The user needs to be able (for testing, ...) toggle gpio pins. There's script for command line control of these pins, but the pin mappings (which pins match which outputs on different boards) is inside the settings table in a database (so accessible with a script in p_connections)

I could use something like

from .../p_connections import credentials

but that doesn't seem very ...elegant.

Another option I see would be to add all my submodule paths to the python path, but that seems overkill.

How is this handled "properly"?

Most online replies that I seem to find are in line with "you're doing it wrong, that's not how you should structure a project", but that's not really helping. I've read a couple of books/articles that suggest to really think about your project structure before starting, but I can't seem to get it right...

Answer 1

If the various modules form a single project, they should be part of a common package and use relative imports (from ..foo import specific_function). For example, instead of this project layout:

my-project/
  scripts/
    ...
  p_foo/
    __init__.py
  p_bar/
    __init__.py

I would expect a layout like this, without any Git submodules or similar:

my-project/
  p/
    __init__.py
    __main__.py
    foo/
      ...
    bar/
      ...

(The __main__.py file allows the module to be used as a command line tool like python3 -m MODULE OPTIONS....)

If these modules should be installed and developed separately as their own packages, they should instead all have a setup.py so that they are PIP-installable, and declare their dependencies on the other packages. It is OK if the dependency is not resolved via PyPI but e.g. through a Git URL. With setuptools, the setup.py can declare entrypoints. These entrypoints will be installed by PIP in a way that allows them to be invoked as ordinary command line tools. By installing in development/editable mode (pip -e), this will not impede your edit–test cycle.

Approaches like setting the PYTHONPATH technically work, but that is only really suitable for self-contained applications that don't have to interact with other packages. But you should not expect that your scripts will only be invoked for a certain working directory in order to resolve imports, so setting the PYTHONPATH (or sys.path) can have merit. In my opinion, the previously suggested mechanism – to let PIP sort out the paths – is vastly superior.

A note on Git submodules: they are great for including some other repo at a local path. This is a good alternative to “vendoring” dependencies, i.e. to literally copying the source code of the dependency. However, you have to explicitly select a commit that the submodule should use – it will not automatically follow the HEAD of a repository. Thus, submodules are most appropriate for referencing stable-ish internal repositories that will be updated occasionally, but are entirely unsuitable for closely related projects that will be developed together. If you expect to regularly have changes that will span multiple projects, that can indicate that you should keep all components in a single monorepo.