7

I'm in need of some advice before going foward.

I want to build several large scale projects, like a marketplace product, and a few domain-specific products and libraries. The products may or may not be related but may share libraries. Each product may very well be their own companies [e.g. the marketplace]).

I'm for using monorepos. There are as many benefits as there are disadvantages. Pro-monorepo points are discussed in Advantages of monolithic version control at danluu.com and in the associated links in the article.

Why I think I need a monorepo — I've read about the pros of monorepo, but I'm mainly focused on following:

  • Sharing closed-source code. The products and projects will share internal (closed-source) libraries. Code sharing becomes simpler because closed-source dependencies are locally available. Project-based repositories require proper tooling for building. I'm trying to avoid this requirement since I'm unable to find any tools for managing closed-source dependencies using multiple repos.

Why monorepos will be a problem

  • Inability to scale.
  • Inability to rollback changes without major problems.
  • Inability to hide sensitive code. Because products and projects share a single repo, code from one product will be exposed.

Questions

  • What's the best practice for managing several closed-source large-scale projects in a monorepo?
  • Would it be wise (and sane) to have these large-scale projects in the same repo?

Thanks guys

Improve this question

2 Answers 2

Reset to default
3

To manage large scale complexity, the winning principle since Julius Caesar is "Divide et impera", in English divide and rule (or divide and conquer). This principle applies to political empires as well as software empires.

Software Engineers do not intend to conquer empires, so they call this differently. The main variants of this principle are separation of concerns, componentisation and encapsulation (in the sense of information hiding, that is hiding the internals of a component). All these principles are based on the fact that it is easier to build independent loosely coupled parts and when needed, assemble the black boxes to achieve higher value without getting lost in details and unexpected interactions and dependencies.

Adopting several independent repos allows you to empower the different participating teams to use the procedures that best fit their needs, to manage different release schedules, and -- why not -- open source, sell or subcontract the different components to leverage efficiency (e.g. open source a more general library but still keep control on your other more specific products).

All this (and in addition the many other challenges related to the monolith) certainly explains why monorepos are nowadays more the exception than the rule. I'd strongly advise you to reconsider your choice.

Improve this answer
2
  • Thanks! Very well explained. My next question is, what tools will I need to manage multiple repos upon reconsidering? For dividing projects into multiple repos, I've been looking at Gemfury (paid) for storing packages privately. Also, I've been looking at git subtrees. Gemfury is best option so far.
    – eurekasfray
    Sep 17, 2017 at 16:16
  • You could indeed consider the use of some package managers to automate dependency management and deployment. I'd avoid debate of which third party service could be best, as this would be very opinion based. But I think that discipline and standardized practice (e.g. semantic versioning and efficient coordination) are at least as important as the tools used.
    – Christophe
    Sep 17, 2017 at 23:23
1

Focusing a bit on what you listed as problems.

Inability to scale

A monorepo is not a single repository, it is a collection of repositories (as many as you want, potentially even using different version control systems) managed together in a monolithic manner so that it appears as a single repository. Scalability from the version control system(s) is not an issue.

Since you're still considering using a single repository I guess you're not one of the few cases in which pulling a workspace with the entire codebase would be problematic due to sheer size. Such cases would require innovative ways of building the products.

Some monorepos have support for sparse operation - a workspace would contain just a subset of the codebase repositories. See also Expanding Contracting Monorepos.

Inability to rollback changes without major problems.

This is not a problem at all due to the monolithic management - simply rollback the monorepo version, which should coherently rollback each of the involved repositories to the correct versions.

Inability to hide sensitive code. Because products and projects share a single repo, code from one product will be exposed.

Again, not a single repo.

And now the questions.

What's the best practice for managing several closed-source large-scale projects in a monorepo?

Donno about best, different people have different opinions :) I speak from an extensive experience with a (custom) monorepo with over a thousand repositories, serving many teams and embedded products (sharing vast amounts of code):

  • discourage independent development in the individual repositories from the monorepo, everything should consistently go through the monorepo, with a good CI/CD system in place operating at the monorepo level
  • best served IMHO with Trunk Based Development

Would it be wise (and sane) to have these large-scale projects in the same repo?

In the same repo probably not. In a monorepo, IMHO yes - you already know the advantages.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.