8

I'm in need of some advice before going foward.

I want to build several large scale projects, like a marketplace product, and a few domain-specific products and libraries. The products may or may not be related but may share libraries. Each product may very well be their own companies [e.g. the marketplace]).

I'm for using monorepos. There are as many benefits as there are disadvantages. Pro-monorepo points are discussed in Advantages of monolithic version control at danluu.com and in the associated links in the article.

Why I think I need a monorepo — I've read about the pros of monorepo, but I'm mainly focused on following:

  • Sharing closed-source code. The products and projects will share internal (closed-source) libraries. Code sharing becomes simpler because closed-source dependencies are locally available. Project-based repositories require proper tooling for building. I'm trying to avoid this requirement since I'm unable to find any tools for managing closed-source dependencies using multiple repos.

Why monorepos will be a problem

  • Inability to scale.
  • Inability to rollback changes without major problems.
  • Inability to hide sensitive code. Because products and projects share a single repo, code from one product will be exposed.

Questions

  • What's the best practice for managing several closed-source large-scale projects in a monorepo?
  • Would it be wise (and sane) to have these large-scale projects in the same repo?

Thanks guys

1
  • "Each product may very well be their own companies" Be very careful here about conflicts of interest and IP infringements. You might expect to run all these companies yourself but from a legal perspective they are different entities with their own sets of rules and responsibilities. It already infringes on the "closed" nature of what constitutes a "closed source".
    – Flater
    Commented Mar 19 at 2:24

3 Answers 3

4

To manage large scale complexity, the winning principle since Julius Caesar is "Divide et impera", in English divide and rule (or divide and conquer). This principle applies to political empires as well as software empires.

Software Engineers do not intend to conquer empires, so they call this differently. The main variants of this principle are separation of concerns, componentisation and encapsulation (in the sense of information hiding, that is hiding the internals of a component). All these principles are based on the fact that it is easier to build independent loosely coupled parts and when needed, assemble the black boxes to achieve higher value without getting lost in details and unexpected interactions and dependencies.

Adopting several independent repos allows you to empower the different participating teams to use the procedures that best fit their needs, to manage different release schedules, and -- why not -- open source, sell or subcontract the different components to leverage efficiency (e.g. open source a more general library but still keep control on your other more specific products).

All this (and in addition the many other challenges related to the monolith) certainly explains why monorepos are nowadays more the exception than the rule. I'd strongly advise you to reconsider your choice.

2
  • Thanks! Very well explained. My next question is, what tools will I need to manage multiple repos upon reconsidering? For dividing projects into multiple repos, I've been looking at Gemfury (paid) for storing packages privately. Also, I've been looking at git subtrees. Gemfury is best option so far. Commented Sep 17, 2017 at 16:16
  • You could indeed consider the use of some package managers to automate dependency management and deployment. I'd avoid debate of which third party service could be best, as this would be very opinion based. But I think that discipline and standardized practice (e.g. semantic versioning and efficient coordination) are at least as important as the tools used.
    – Christophe
    Commented Sep 17, 2017 at 23:23
0

Focusing a bit on what you listed as problems.

Inability to scale

A monorepo is not a single repository, it is a collection of repositories (as many as you want, potentially even using different version control systems) managed together in a monolithic manner so that it appears as a single repository. Scalability from the version control system(s) is not an issue.

Since you're still considering using a single repository I guess you're not one of the few cases in which pulling a workspace with the entire codebase would be problematic due to sheer size. Such cases would require innovative ways of building the products.

Some monorepos have support for sparse operation - a workspace would contain just a subset of the codebase repositories. See also Expanding Contracting Monorepos.

Inability to rollback changes without major problems.

This is not a problem at all due to the monolithic management - simply rollback the monorepo version, which should coherently rollback each of the involved repositories to the correct versions.

Inability to hide sensitive code. Because products and projects share a single repo, code from one product will be exposed.

Again, not a single repo.

And now the questions.

What's the best practice for managing several closed-source large-scale projects in a monorepo?

Donno about best, different people have different opinions :) I speak from an extensive experience with a (custom) monorepo with over a thousand repositories, serving many teams and embedded products (sharing vast amounts of code):

  • discourage independent development in the individual repositories from the monorepo, everything should consistently go through the monorepo, with a good CI/CD system in place operating at the monorepo level
  • best served IMHO with Trunk Based Development

Would it be wise (and sane) to have these large-scale projects in the same repo?

In the same repo probably not. In a monorepo, IMHO yes - you already know the advantages.

1
  • "A monorepo is not a single repository, it is a collection of repositories (as many as you want, potentially even using different version control systems) managed together in a monolithic manner" This is plain incorrect, the definition you're providing here is one of a polyrepo (by the very definition of the word) whose deploy process is that of a monolith. Monoliths act as a deployment process, not as a versioning process. Monoliths are unrelated to how you version your code. Missing the mark on defining core concepts (monorepos and monoliths) calls the rest of this answer into question.
    – Flater
    Commented Mar 19 at 2:29
-1

Yes, it would be absolutely a good idea to collocate several large-scale projects within a single monorepo, because that immediately removes obstacles for reuse, maintenance and collaboration, and things that change together should go together.

Since you're in the best position to understand what your needs are, and given how you've stated the post, you already have the right answers to your questions, and I am assuming here what might be missing in your case is just resolving any remaining uncertainties and working out the details.

I can affirm that, and also offer the most challenging case for your solution, which should hopefully give you an idea of how you can best approach these questions.

Is this sane?

When you have multiple repositories, what makes them distinguishable from one another is that each has a unique base path, e.g. acme/frontend, acme/backend, etc.

Collocating everything under a single monorepo, say acme/omni, is just a slight shift of appending a common base path to all the changes therein.

To state the obvious, code doesn't suddenly lose the ability of being uniquely qualifiable. So if you think of this as a more general relationship, what holds true for code holds true for X, then and now, then, by virtue, you can still have independent code ownership, builds, deploys, rollbacks, issues, pull requests, project boards and everything else that's already available to each project on it's own, on the platform you're collaborating on.

To be realistic about these things, note the fine point on everything that's already available. If you cannot have some people see and not see some part of a project's repository they're working on, as is usually the case, then you can't expect to have that in a monorepo either.

What are the best practices?

Essentially, do as you would to make each project a success, except now there are no immediate technical limitations to make changes in a manner that addresses the needs of all the projects together.

So, if you went down this path because you realized there's things that would be good for more than one project, you already are the kind of a person that can look at these efforts as being best addressed form a unified whole, and that's the only thing that matters.

Hard case: some projects will need to exist outside the monorepo

What makes it hard to have a project existing in a separate repository is that everything you've tried to avoid with the simplicity of having a single monorepo now needs to be managed.

However, what gives an upper hand to someone tackling this from a monorepository is that everything within is already neatly modularized and disaggregated, due to the lack of technical restrictions, so it's a matter of adding support for such integrations on top of a sound base, whereas someone who finds themselves in a zoo of repositories needs to go through all the modifications to introduce sanity beforehand.

This is a long list of things and all of them are non-trivial, but basically everything tangible from your monorepo, the culture, chores, infrastructure code, build configurations, common libraries, services, assets and so forth, now need to be set up in a way that they can be made accessible to remote repositories.

In this, when it comes to everything that falls under VCS in particular, aside from the more traditional approaches of using some form of native capabilities to reconcile changes happening separately between independent repositories, or just degrading back to naively copy/pasting things all around, emerging tools like projen can be used. Much in a way a compiler can bootstrap itself, a monorepo applying projen onto itself can self-bootstrap, then proceed to automatically manage other repositories.

Personal musings on the matter

Good practices should not and cannot wait months or years to propagate and come into effect. This is mandated by the outside world, and so modern-day technologies already come out with support for all members of a single organization to work together in a way that when one unit strikes gold we are zeroing on having this instantaneously available to everyone else.

The monorepo is one such device, a very simple and effective one, one of many, and yet there's still a prevalent misconception that we absolutely need to have repositories such as acme/frontend and acme/backend.

This is not true, and if done so it should be only because it is instrumental, because it cannot be done otherwise, legal reasons being one good example. Making claims that there are prohibitive technical issues, such as "it's not going to be possible to do X separately in a monorepo, like we can now" in these cases is a false dichotomy. I find it a tough pill to swallow with how we still seem to love our islands and silos, while paradoxically we are giving it our all to give everyone the best support they need to do their job right.

2
  • 1
    This answer makes broad-sweeping global assertions that lack nuance and context. Whether or not FE and BE should be independently versioned/deployed or not is (a) not universally answerable and very much dependent on business context which is sorely lacking from this answer, but more importantly (b) does not answer whether or not different applications (to the point of belonging to different companies) should reside in a monorepo or how they should otherwise handle any shared logic between them.
    – Flater
    Commented Mar 19 at 2:32
  • Thanks for the feedback @Flater! I've made changes to the initial answer and tried to fix some of my mistakes, make it look less like a personal rant. Commented Mar 22 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.