3

I currently am working on a project that involves several discrete components that have some common functionality, and I am in need of some advice.

Right now I have a central component responsible for dispatching requests to several components that translate the requests to particular formats for particular data sources. The components are running as separate web applications behind an Nginx server. Communication between all the components is via REST.

One issue I'm running into is the fact that many of the components need to perform similar tasks, necessitating code duplication, which I want to avoid.

For example, the data-source specific components use credentials to authenticate to their respective data sources, but the central component also needs access to this store to add new credentials and update existing ones (credentials are stored in a common database).

Right now each component is its own project/app, in it's own separate folder.

If this was a Java project, I would immediately pull out the common functionality, create a library JAR, and have all the components reference it.

However, I'm not quite sure how to do this in Python. I can't really combine the components into one monolithic unit because I need the flexibility to add new components for new data sources (unless somebody knows of an easy way to handle this). Also I eventually want redundant components to handle heavy loads.

First, should I take out common functionality and put it into it's own project/library at all? Is it worth it? If so, how is this usually done in Python (I'm using version 2.7)?

Do I need to create a distributable, (using pip for instance), or is there a way to define a common library without going through that trouble?

2

In Python, we have packages and modules. A module is "just" a .py file, and a package is "just" a directory with an __init__.py file (which is often empty). You can import any package or module which is in a directory listed in sys.path, modules within such a package, subpackages within such a package, modules within such subpackages, and so on.

While you can put the library in one of the standard system package directories, this is often more trouble than it's worth, since you'll be making the library available system-wide. You can also put the package directly in the project directory, but this is somewhat fragile depending on how your app gets launched. In such circumstances, we commonly use the Virtualenv tool. This allows each app to have its own separate package directory.

Note that the import system was overhauled in Python 3, as described in PEP 328, and is now somewhat more consistently behaved. Python 2 users can get the new behavior with this:

from __future__ import absolute_import
  • I actually am using virtualenv for this setup already, so I can just add one more package here. I am new to using it, but I assume it is just a matter of realize that the sys.path in this case is within the virtualenv directory, correct? – Chris Chambers Jul 31 '15 at 20:02
  • Yes, the virtualenv will generally put its own directories into sys.path and (normally) remove the system and user site package directories. – Kevin Jul 31 '15 at 20:03
  • I assume I can just symlink the folder in the .env/include right? – Chris Chambers Jul 31 '15 at 20:13
  • I think that should work. If not, post on SO. But you might be better off copying it, because that way it won't break every time someone breaks the build in your original folder. – Kevin Jul 31 '15 at 20:15
  • Will do, I'm going to premptively ask this on SO because I have a bit of a strange setup and I foresee so many problems if I don't do this just right. I'll link it here in case anybody is interested. – Chris Chambers Jul 31 '15 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.