How to manage multiple projects in a Git Monorepo

Question

I am thinking about putting some files into a larger git respository but I have a feeling that git isn't going to be suitable.

This is what I would put into the repo:

1. Various scripts on servers.
2. Some larger projects in var/www.
3. Some larger projects elsewhere.

All of these programs at some point talk to each other and call each other, because of this I thought that a monorepo would be a good idea.

However some possible issues:

1. How to check out certain directories? Especially on other servers.

For example, what if I want to check out a single script onto another server from git, I don't want to have possibly 500mb of code sitting on that server too.

If not git, is there another version control that will better suit my needs?

I've considered changing our development pipeline, for example, the repo could live on a dev server, and we use rsync scripts to push files to and from other servers.

What I am looking for is possible solutions and peoples experience with this issue.

To be more precise, my issue is that I currently manage some 20-30 smaller repos and I feel that jumping around these repos isn't very efficient.

The key point is that we have so many programs over different servers and many folders inside those servers, but all of this code is essentially apart of one large system.

Answer 1

Are you aware that you can checkout only the latest version? https://stackoverflow.com/a/1210012/1177905 that at least removes all history so you get less deployed.

Then about your repositories: If they are all part of one solution with one deployment I would group them in one repository maybe.

An alternative would be to create a new repository per solution. So let's say you have libraries 1-30 (your current repos). You create a new repository for web service and for mobile app.

The only thing those repos do is just pulling the solution together:

Web service

• Library 1
• Library 2
• Library 3

Mobile app

• Library 2
• Library 3
• Library 4
• Library 5
• Library 6

etc. So these new repositories combine all stuff to get a complete release. So your deployment pipeline (and for example end-to-end tests) only build those repositories. You will need a central place to test and deploy the solution anyway.

That keeps you more flexible and you can be sure that a fixed set of libraries on specific versions gets deployed. And it prevents that you have to change all libraries.

If the repos are not consistent you might want to structure them a bit to become more equal in interface. That will further improve your codebase. Microservices do this a lot, they have some general way of calling each other so you don't have to think about that every time in a custom way.

Answer 2

The issue here appears to be that you're conflating your system for releasing production code with the system for developing that code. Git is a great tool for a developer or build system to grab all the source and start building and testing. It isn't always the best tool for deploying that code once you've confirmed that it's working.

Have your build system (whether automated or manual) build the individual products you want to deploy in whatever way works best for that particular deployment. Here are some examples:

• Push an individual file out to the production server via scp.
• Push a set of files out to a staging point (e.g., a web server, a shared drive, an Amazon S3 bucket) and have the server pull those.
• Build a tar file of the necessary files and push that to a server or staging point; the server then unpacks them after getting them. (If you keep the older versions of the tarballs this can make rollbacks easy.)
• Build a package for an appropriate packaging system (Python package, Ruby Gem, Debian package, RPM) and distribute that.
• Build a Docker image that is copied to the server or a central repo whence the servers pull it.
• Have some servers continue to pull the whole repo, if they use a lot of files in it and that's easiest.