-2

Considering a Python Project structure such as the following, where there are "empty" packages with __init__ files simply pulling code from the lib folder:

.
├── foo
│   ├── fred
│   │   └── __init__.py
│   ├── henk
│   │   └── __init__.py
│   ├── _lib
│   │   ├── classes
│   │   │   ├── corge.py
│   │   │   ├── _stuff.py
│   │   │   └── thud.py
│   │   └── utils
│   │       ├── bar.py
│   │       └── _stuff.py
│   └── __init__.py
└── script.py


# foo/__init__.py:
from foo import henk, fred
# foo/fred/__init__.py:
from foo._lib.classes.corge import Corge
# foo/_lib/classes/corge.py:
from foo._lib.classes._stuff import *
class Corge():
    pass

The reason for this is, while a bit unorthodox looking, it seems to help with code autocompletion. Irrelevant/internal modules from a package don't show up in tools tips, like such in Spyder IDE: enter image description here

For context, I didn't want _stuff or thud showing up at that level; this file structure achieved this.


It's been working so far, and so I've been wondering if there are any potential side effects since I've never seen this structure before? Could this be unfriendly to contributing developers or their tools?

Any other ways to achieve a similar goal (not clutter namespace for users) would be very welcome.

1 Answer 1

0

Yes, this is sensible. And no, this is not the correct way to do things.

The pattern where you have a top-level module that re-exports symbols from private modules is somewhat common for larger projects. However, the main selling point for these modules is that your public API is defined in a single place, not that this would help to hide implementation details.

To control which symbols are intended to be accessed from outside a module, the following techniques are used instead:

  • If something should be internal, give it a name that starts with a leading underscore. Python has no C-like reserved names that could interfere here.
  • To control which symbols are exported by default, assign their names to the __all__ list. Technically this is just the default export list that is used for from module import *.
  • To some degree, you can del variables that are no longer required. However, this requires a precise understanding of what is evaluated when.
5
  • Sadly I've been unable to implement the __all__ list successfully, as I'd like just the Class to show up autocomplete (as shown) instead of module + Class. The leading underscore filenames still show up for autocomplete in the target IDEs which is why they've been tucked away - If you could provide a mini tree structure so I can see if I'm missing something that'd be very helpful; though I'll keep trying to figure it out and update accordingly.
    – lucasgcb
    Feb 18, 2019 at 8:27
  • @lucasgcb I really understand why you find that autocomplete behaviour frustrating, but I don't think you should design your programs around a particular IDE. Maybe your autocomplete engine has a configuration option to exclude private module members, or to at least show them with lower priority?
    – amon
    Feb 18, 2019 at 8:54
  • I do agree with that! However, the behaviour for the IDEs are a requirement for this project as the library will be used by scientific workers (who use Spyder/Pycharm); they want the least cluttered, most straightforward autocomplete for this toolkit they can get. Since it is "sensible" solution for the client, I don't really have an argument against it besides "it is not a standard technique". That's why the question ponders after "possible issues" in implementing such an unorthodox structure for the most part. I've considered IDE configuration, but was unable to find anything useful.
    – lucasgcb
    Feb 18, 2019 at 10:41
  • @lucasgcb in that case putting your public API into a top level module is sensible so that users can do from foo import x, y. In your example this would be the foo module. But I don't see the value in using this for modules that will only be used internally, or to create many helper modules that re-export the variables (like fred and henk in your example).
    – amon
    Feb 18, 2019 at 12:59
  • Alright, I'll mark this as resolved then. I'll update if I ever manage the solution that complies with the structure standard and achieves the desired autocomplete declutter for the target IDEs. This could be a Y problem since the real issue is just hiding junk from autocomplete, so thanks for the patience.
    – lucasgcb
    Feb 18, 2019 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.