Project structure with git submodules with common dependencies

Question

I have two (and more in the future) Python projects that I'd like to package into a global package in order to import its modules and use them in another external projects in the future.

This is how it is structured now:

I made it using git submodule, so I made that Project 1, Project 2, etc. were submodules of the global project.

Now I have a problem, because in Project 1 I use a Python module (a .py file) that I need to use in Project 2 too, and probably in all N projects.

Should I make another git repo that contains all that useful modules to import in other projects, and add to all them as a submodule?

Example:

Or can I somehow make that the global project has a library and import the files from there? Here we have a problem, because each project separately will not have the files, so they won't be able to import them

I also was thinking about creating a git repo that has all projects inside, but it wouldn't possible since there are different people working on each project, and they shouldn't be allowed to access to all the repository

Answer 1

Using submodules for this is fine... but can be suboptimal, depending on what your priorities are, and what you value most. Think of git submodules just as one of the possible tools at solving a series of problems. There are some classic alternatives to it, like git-subtree (now it's part of the regular git package), or repo. Some other domains have domain specific tools for this. E.g. for Yocto related projects, there are tools like kas, combo-layer, etc. (though using repo and git submodules is also popular).

For the kind of problem that you have, you might want to look instead at gil (git links). It's a less known (I think) tool, but it's written in Python, and roughly tries to solve the same problem as yours.

I personally I'm not a fan of some consequences that gil has (the "rolling release" approach), and I still use git submodules, even if in some projects that might be some duplication. I don't think that bundling one project into another it's the right approach anyway. ProjectA and ProjectB can depend on HelperFoo if the latter is added to a shared location. If the projects depend on a different version of the helper, then a naming convention or a prefix can give you enough in any case that I would consider sane enough (or locally bundling one specific case, if it makes sense, but not as the preferred approach to attempt first. Linux distributions have been following this approach for ages anyway.