Structuring a Python Project: Need some guidance

Question

I currently am working on a project that involves several discrete components that have some common functionality, and I am in need of some advice.

Right now I have a central component responsible for dispatching requests to several components that translate the requests to particular formats for particular data sources. The components are running as separate web applications behind an Nginx server. Communication between all the components is via REST.

One issue I'm running into is the fact that many of the components need to perform similar tasks, necessitating code duplication, which I want to avoid.

For example, the data-source specific components use credentials to authenticate to their respective data sources, but the central component also needs access to this store to add new credentials and update existing ones (credentials are stored in a common database).

Right now each component is its own project/app, in it's own separate folder.

If this was a Java project, I would immediately pull out the common functionality, create a library JAR, and have all the components reference it.

However, I'm not quite sure how to do this in Python. I can't really combine the components into one monolithic unit because I need the flexibility to add new components for new data sources (unless somebody knows of an easy way to handle this). Also I eventually want redundant components to handle heavy loads.

First, should I take out common functionality and put it into it's own project/library at all? Is it worth it? If so, how is this usually done in Python (I'm using version 2.7)?

Do I need to create a distributable, (using pip for instance), or is there a way to define a common library without going through that trouble?

Answer 1

In Python, we have packages and modules. A module is "just" a .py file, and a package is "just" a directory with an __init__.py file (which is often empty). You can import any package or module which is in a directory listed in sys.path, modules within such a package, subpackages within such a package, modules within such subpackages, and so on.

While you can put the library in one of the standard system package directories, this is often more trouble than it's worth, since you'll be making the library available system-wide. You can also put the package directly in the project directory, but this is somewhat fragile depending on how your app gets launched. In such circumstances, we commonly use the Virtualenv tool. This allows each app to have its own separate package directory.

Note that the import system was overhauled in Python 3, as described in PEP 328, and is now somewhat more consistently behaved. Python 2 users can get the new behavior with this:

from __future__ import absolute_import