Fork Management of Git Multi-level Hierarchies

Question

We have the following Git hierarchy structure in place, where each child is forked from the parent. If we build something in Client1 that will be useful in the entire hierarchy, what is the best way to push the changes up to the parents (whether just one level up or all the way to the root), given that we don't wish to push all the Client-specific customizations up the chain.

Now, while the ideal solution might be to do the development at the highest required level, the modules can only really be tested down the chain where all the customizations are instantiated.

E.g. If we develop a Module-New in Client1 repo, and need to push it to Core, what would the best way for that to be done given that we don't want Client1's customizations in the core? On the other hand, if we had originally developed ModuleA directly in CORE, we would have had to push the changes downstream all the way to Client1 to test Module-New (and iterate for every change).

Are there any Git Workflows or any other tools that can help establish these rules. We currently use GitFlows to manage our master, develop and feature branches for all repos. Also, is there a way to disable direct pushes into a remote repository?

                            CORE
                              |
                ------------------------
            Product 1               Product 2
                |                       |
            ----------              ----------
        Client1     Client2     Client3     Client4
           |-v1.0   
           |-v1.1
           ...

Answer 1

You can disable direct push in repositories depending on which technologies you're using. Preventing push can be done somewhat using git hooks, but if you're using gitlab/github/bitbucket... there should be a way to protect branches.

If all branches are protected and that the dev can't create new branch, then there is no way to push to the repo. This way, you'll have to handle the dev using "merge requests/push requests" or how it's called.

It's unclear what you're actually managing so the following might not help. In my case, we have a core product that is installed on every client, the core product might be installed with different versions. And additional modules can be installed on the side. Either way, usually client customization are usually done by installing specific modules for clients that extend existing modules (including core modules). In my case, customizations for clients never or rarely bubble up to the core product. If we needed to, we could simply move the module into the core product.

If a custom module is shared across multiple clients, we might use special branches that are merged on the master of each module repository.

In the end, it can get quite complicated if code doesn't bubble up to the master branch of each module's repo. In our case, most of the work is achieved using buildout to manage repos and merges and dependencies.

My opinion

Handling dependencies using git repositories should be limited to development even if it looks simpler to just fetch the repos. Git repository can get quite big and sometimes fetching might fail if the repository is big and the internet connection is unstable...Git doesn't allow partial fetch so if it fails at 99%, you'll have to restart from 0% anyway.

For deployment, I think it makes more sense to have package managers do it. They exists and they usually do their jobs well. The advantage of using a package manager is that you don't have to download the whole history of changes that might come with the module. Usually package managers allow partial download using http so you can pause an update later if internet fails. With a good package manager, you could also host your own private package servers for each client. Any module that is similar for each client can be pushed on a central server and every customized package could be pushed to a package server per client.

All of this is fine if you have a way to automate the building process of package and deployment of package on their respective servers.

This way if a client does something like "package update", it could check on the client server for a package and then on the central server and take the right version from this. Old package don't have to be deleted and the client could downgrade its setup as he wish.

My guess, is that it's the nicest thing someone could implement, yet there is no standard way to do that as far as I know. I heard people doing that with python eggs, other with debian packages an so on. But each implementation has their own problems... Like debian packages won't work on windows and non debian distros... pythong eggs are quite limited to python.. npm packages to javascript and so on. If we had a universal package manager that would be simply a piece of cake to implement.

Answer 2

Managing per client code customisation with source control is not a good idea. It's not what source control is designed to do.

In your case for example, there is no way to determine whether the 'new module' is dependant on the customisation in the branch. so you cant just merge it up and expect it to work.

The correct approach is to split the project into components and create a new customised component for clients. When deploying a client instance, you use DI and configuration to specify that the custom component is used rather than the standard one.

All the components are separate repos and published via a package manager so they can be pulled into other projects

If common functionality is used across customised components, these are again split so that the common code is in its own module and the customised components reference it. So in your scenario of wanting to move code 'up a level' you would edit the common component to add the feature to it, and edit the custom component to remove the feature from its code, but reference the new version of the common component which includes the feature