1

I want to structure my Python project in a manner such that others can use a package manager (PIP) to easily clone the environment for team development, and GIT to contribute to the project, plus a library dependency. My confusion relates to organizing a project around PIP and GIT (I'm familiar with SVN).

The project depends on a library from GitHub that I forked, which I'd like to have appear within the project source tree.

Devs on our team should be able to

  • easily clone the environment using a Python package manager
  • see the library dependency code in the source of the project (whether it's pulled from GIT, or PIP - am unsure of best practice)
  • be able to commit changes to project
  • be able to commit changes to library and create a pull request

What's a good approach to this? I tried pip install -e git+https://github.com/nxorable/FuelSDK-Python.git@v1.0.0#egg=fuelsdk to install the library dependency from source. This leads to a detached head on that library's folder, so that I can't commit changes. I suspect that means I'm not doing something right.

  • Not sure if I understand correctly: you want people to be able to contribute to the dependency (which was not created by you but which is part of your project) but not to your own project? So, your project wouldn't use git but you'd like people to know they could pull changes made to the original dependency into their copy of your project (which they got via PyPI) and also submit own code to it? – Kay Jun 19 '16 at 14:36
  • Yes, I'd like to allow my team to contribute to the dependency, which was not created by me but is part of the project. – Brian Bien Jun 20 '16 at 1:32
  • My project would use GIT – Brian Bien Jun 20 '16 at 1:37
1

Don't think too much about Pip. If your Setuptools configuration works correctly, it's easy enough to upload all and only those files you actually want to PyPI (at which point end users will be able to install it with Pip; developers will be cloning your git repository since they need history for things like git bisect to work, and Pip does not provide git history).

Worry instead about how your project will be laid out. One common layout is something like this:

 Root directory
|
+- name-of-your-project/
| |
| +- __init__.py
| |
| +- other modules and packages...
|
+- name-of-forked-dependency/
| |
| +- __init__.py
| |
| +- other modules and packages...
|
+- setup.py
|
+- Other stuff...?

"Other stuff" might include documentation, unit tests, and other miscellaneous materials which you do not want to distribute alongside your code. It is also somewhat common to put a requirements.txt file here as a hint to developers that they should install certain packages in order to work on your project. Note that Setuptools is smart enough to include some files but not others, so you can just put most of these files directly alongside the source code if that is your preference.

The next question is how you get the repository to look like this while preserving the history of your forked dependency. Assuming the other project follows the Python standard of keeping its own code in a directory named after its root package, you have two options:

  1. If your code already exists and has history, you may want to create a git repository with two roots. One root will be the initial commit of your code, and the other will be the initial commit of the code you are forking. So far as I can tell, this appears to be doable by simply doing a git remote add and then a git fetch on the new remote. Afterwards, you probably want to merge the branches together with git merge, at which point you will have a single master branch with all of the history from both repositories.
  2. If your code does not yet have history, this is even easier: Just clone the repository you want to fork, change the origin to something sensible such as a newly-minted GitHub repository, push, and begin development.

The above assumes you want to distribute your forked copy of the code as a unit alongside your actual project. It may be more sensible to distribute them separately and list the fork as a dependency of your code in the Setuptools configuration. That way, you can maintain two separate repositories for each of them, and provide a separate package on PyPI for people who just want to work with your forked code. Developers who want to contribute to both repositories will have to do slightly more work, however, so you will need to judge the benefits and drawbacks yourself.

For completeness, you should also be aware of the Virtualenv tool, which is invaluable for managing complex dependency situations such as yours, and is also quite suitable for even the simplest of deployments.

  • Thanks, Kevin. As it turns out, I'm using conda instead of virtualenv (I thought it would detract from the main question, so left it out). – Brian Bien Jun 20 '16 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.