I currently am working on a project that involves several discrete components that have some common functionality, and I am in need of some advice.
Right now I have a central component responsible for dispatching requests to several components that translate the requests to particular formats for particular data sources. The components are running as separate web applications behind an Nginx server. Communication between all the components is via REST.
One issue I'm running into is the fact that many of the components need to perform similar tasks, necessitating code duplication, which I want to avoid.
For example, the data-source specific components use credentials to authenticate to their respective data sources, but the central component also needs access to this store to add new credentials and update existing ones (credentials are stored in a common database).
Right now each component is its own project/app, in it's own separate folder.
If this was a Java project, I would immediately pull out the common functionality, create a library JAR, and have all the components reference it.
However, I'm not quite sure how to do this in Python. I can't really combine the components into one monolithic unit because I need the flexibility to add new components for new data sources (unless somebody knows of an easy way to handle this). Also I eventually want redundant components to handle heavy loads.
First, should I take out common functionality and put it into it's own project/library at all? Is it worth it? If so, how is this usually done in Python (I'm using version 2.7)?
Do I need to create a distributable, (using pip for instance), or is there a way to define a common library without going through that trouble?