First off please note, I’m fully aware that Git has historically been terrible at dealing with large binary files and very large repos; there are at least a dozen questions on SO that deal with large dataset performance, but as far as I’ve found, they’re all years old and the answers all more or less predate a number of important contributions by Microsoft and others to drastically improve Git’s scalability, such as partial clone and enhancements to sparse checkout.
Context
I want to be able to put a whole game project in a single repo if I possibly can, because the programming and assets have a degree of coupling, there are various tooling scripts and metadata that accompany assets, and having to keep track of a number of separate repos for one logical project seems like a pain. The programming specific to the game is relatively simple script; the game development system and engine which are common to multiple games will be in separate Git repos.
(Full disclosure: I am totally new to revision control. I somehow managed to come up with a whole game engine and development system, mostly single-handedly, using only routine backup plus the occasional copy-paste of source folders.. lol.)
Characteristics of the contemplated project monorepos:
- Total data size of revision controlled files (i.e., only those that aren’t reproducible from other files) in the project tree is expected to be 100 ‑ 250 GB for a smallish project, probably into terabytes for a larger project. The one I’m trying this project monorepo thing out on first will be a smallish project. This doesn’t count revisions, but see below. I may be able to exclude some types of authoritative files to get that size down quite a bit, but doing so would trade off convenience and safety, and I’d really prefer most authoritative data is in the repo.
- Very roughly, the number of revision controlled files would likely be 10,000 to 40,000.
- Some binary files go through numerous revisions while others are write-once. The really big ones—i.e., into 100s of MB or more—tend to be write-once, because they tend to be raw recordings which are edited in a non-linear fashion with edit decisions saved into relatively tiny project files. I’ll probably keep such video recordings out of the repo entirely, but in the case of audio it’s harder, because DAW software likes to number recording files on its own, so pesky naming conflicts could arise if those files aren’t checked in.
- Some editable binary files, typically in 5 ‑ 50 MB range, can actually delta compress very well. I’ve seen it claimed on SO that real-life binary files don’t delta compress because they’re compressed. This is an overly simplistic view of actual DCC file formats, where typically compression is done separately on individual sub-objects within the file. I did a test on a set of 10 backup files taken from an actual art project done in Photoshop, saved to LZW compressed layered TIFFs, where I was painting in stages using many layers. By simulating delta compression using LZMA with a large dictionary, I was able to drop 129 MB down to 29 MB, whereas the same LZMA run separately on each version only got it down to 122 MB.
Other Software Considered
The present question is about native Git, but for completeness:
I’m actually also considering Subversion for the game repos. The main drawbacks of SVN are poorer handling of merging and potentially more stress on the central server. Another disadvantage for me would be having to learn two VCSes instead of one. The really big drawback of Git, if it still applies, would be poor scaling on large binary datasets. Other perks of SVN are path-based access control, versioned free-form metadata, and monotonic revision numbers. Plus, I hear Git scares the crap out of artists, so I may need to whip up friendly workflow front-ends.
I’ve also investigated git‑lfs
, git‑annex
, git‑media
, git‑fat
, git‑fit
, and git‑bin
, but I have big concerns with all these:
- Each has implementation and/or maintenance issues. For instance, the only open-source server implementation for
git‑lfs
is not production-ready, so I can’t deploy a local server without buying into some proprietary solution. - I’m going to venture a guess that none of them implement any kind of delta compression, but some binary files in practice can actually delta compress rather well. (I could just use LFS on the really big, write-once stuff, but I still don’t love the idea.)
- They all bake administrative decisions about storage into the immutable history of the data itself. This is conceptually ugly and also makes things brittle.
If it came down to having to use an LFS type extension, I’d almost rather go with Subversion.
Analysis
In trying to figure out whether Git could handle such a game monorepo ok, I analyze as follows: The number of files is not that huge considering some of the larger Git source repos that work just fine. The main thing is file size, which means I have to look for those situations where Git does operations that are O(size) or worse.
For clones and checkouts, it’s of course unavoidable, but nowadays Git can do both of those sparsely, so the client is only pulling what they actually need. So where else could large files choke Git?
There is a ‑diff
attribute which can tell Git not to generate diff text. We should set this on all binary files, since diffs will generally be meaningless, at least the way Git would report them.
There’s also a ‑delta
attribute which stops Git from delta compressing. This is trickier, because as mentioned, some medium-sized binary files actually can be delta compressed quite well sometimes. But delta compression requires time and memory.
It seems Git packs using a sliding window, by default of 10 heuristically similar objects at a time. 10 files of 50 MB each is 500 MB of RAM, which is not totally unreasonable, but could be a slow operation. Just how slow? I imagine it would be advisable to totally disable automatic garbage collection, instead running git gc ‑‑auto
manually when convenient, since it could otherwise happen on any given command, which could be quite disruptive.
Microsoft’s Scalar apparently does some repacking in the background, but that’s only documented somewhat vaguely and in the context of fetch and checkout, so I’m not totally clear on what the implications are on the central repo. (I also hit a bit of research fatigue here.)
Since delta
and diff
are separate attributes, I assume it’s possible to have some subset of files for which Git won’t attempt to difference for history purposes but will try to delta compress. But Git is infamous for being counter-intuitive, so I have to ask. With some files, this may be an appropriate strategy.
Another not-so-clear matter involves the fact that Git tracks content and not files. So, if, for instance, you move a function body from one source file to another, Git can tell you that and also can merge properly across that. But this comes at a cost, which must be at least O(size) in the involved files. To make matters worse, Git stores the snapshots “dumb” and then does this tracking at the time of the query or operation. I think that was a good design decision overall, but it does mean that the O(size) cost may, if I’m not mistaken, get incurred over and over again as you run routine commands.
I wasn’t able to determine whether this “content tracking” is elided on files having the ‑diff
attribute. That attribute officially speaking controls whether diffs are printed, not necessarily whether intra-object content flow is being tracked. This seems to be a very important distinction with very large files.
To be clear, I don’t mean Git shouldn’t track those files at all. I mean instead that the first byte it sees that doesn’t match is enough to just say, “hey, they’re different” and be done with it, rather than trying to figure out exactly what moved or changed where within the files. This would lead to degraded behaviour if a file was moved and changed in the same commit, but that’s tolerable and no worse than SVN. Of course, it’d still take the SHA hash, which is ok as long as the file has been touched.
Question(s)
My core question is, can native Git today reasonably handle what I’m contemplating throwing at it?
I would almost say Microsoft has already answered this question by hosting the Windows OS and MS Office repos in Git, but those are not exactly the same scenario. They’re dealing with a mammoth number of relatively small files, whereas I’m dealing with a moderate number of sometimes very large files. They’re also using enhancements that have been partially but not completely contributed back to the upstream Git project.
That core question leads to some ancillary questions, hinted above, which I’d have almost posted separately, but figured to instead include in context so they don’t become “X/Y” problems.
While this is intended as a question about Git, not a “shopping” question, if some other VCS would be much better in this application, I’m open to that also.
⠀
⠀
Addendum & Update
First, to address @DocBrown and @amon:
⠀
Number of users: Right now only a few, but given migration probably being painful, I wouldn’t want to deploy something that couldn’t grow to, say, 50 total users before having to revisit.
⠀
Git intrinsically suboptimal for large data: This is a big concern and as such I’m leaning towards SVN, which seems more inherently scalable in this regard; see below.
And there is indeed something to be said for not trying to push Git to its limits as a first-time user. With that said...
⠀
Usefulness of sparse checkout: Typically, for a given task, a user could get by with a checkout in the 200 MB to 4 GB range. That’s still a little big for Git, although I also have to note that Microsoft’s Derrick Stolee has mentioned users with 30 ‑ 50 GB of background-fetched packfiles on the client side. It's not ideal to have to explicitly limit the checkout, but that's hard to avoid regardless of VCS.
⠀
Needing VFS for Git: Microsoft’s solution did originally hinge on a virtual filesystem, however this has changed more recently. For the MS Office repo with MacOS users, they developed a more native Git solution, implementing improvements to partial clone and sparse checkout, much of which they’ve fed upstream. See Git at Scale for Everyone.
It is worth noting that many recent scalability improvements to Git are not turned on by default. These settings are still obscure enough that MS has created a tool to configure Git.
The downside to this newer system is that you have to explicitly ask for the tree subset you need checked out, whereas VFS enabled a more transparent experience where it looks like you have the entire repo checked out, with the files you actually use being fetched behind the scenes.
One big concern is that, although you can reduce the amount of data on the client, you still need an authoritative repo somewhere that’s got the whole hundreds-of-gigabytes tree. If you use Azure DevOps Server for that repo, I’m guessing it’d handle ok, but that’s fairly expensive and proprietary. Using native Git for the authoritative repo could get extremely painful depending on exactly when and where it does processes that are O(size) or worse.
⠀
Keeping source code separate: I already intend to partly do so, as the engine, script class libraries, and tools will all be separate, code-only Git repos. This is the reusable “heavy lifting” code. But each game also has script code that is specific to it, though fairly lightweight. I really don’t want to keep that and the assets under separate version control, as they have significant coupling. I’d actually rather not use Git for that code, if it means having to version it separately from the assets.
⠀
Not using a VCS on binary assets: At least some of the binary assets do belong under version control. From a programmer's perspective the assets “just show up” and as such seem like static content. However, I do asset production as well, and from that perspective, many of those files are far from static. I've got directories littered with dozens of haphazardly version-numbered asset files that wouldn't be there with proper version control.
⠀
Present Thoughts
I’m mainly focused on Git vs. Subversion. (I’ve also superficially looked at Mercurial and Perforce.) It looks like it’s mainly a trade-off between Subversion’s apparently terrible merging and Git’s not-so-great scaling to large data.
This question’s premise had been to benefit from Git’s better branching model by relying on recent improvements to avoid its scaling issues, but I’m at this point again leaning more towards Subversion, on the thinking that Git scalability is too risky. Taking a step back I appreciate that scalability is actually much more important to the game repos than ease of merging—getting more merge conflicts would be an inconvenience, but the VCS grinding to a halt would be catastrophic!
Our games tend to be story-based, with most of the boiler-plate mechanics handled in the class libraries and development system, which are not part of the game. The code that actually belongs in the game repo is lightweight and only needs to be touched by a very small number of programmers. Furthermore, a fair bit of it is split up into source files specific to game regions easily written by a single programmer.
This all seems to suggest that code merge conflicts in practice could be fairly minor even if the VCS isn’t that great at merging. OTOH, experimental branching would actually be more useful for the game than for the development system, so SVN isn’t ideal, either.
It feels a bit strange contemplating taking up a tool that much of the software industry has abandoned and whose users Linus Torvalds has called ugly and stupid. Almost like investing in dinosaurs at the end of the Cretaceous. But yet I have to wonder if it might actually be a better tool for this job.
I’m going to read the manual for SVN so I can ask questions about it more intelligently before deciding. But if anyone can say confidently that Scalar actually will be fine in this use case, that‘d be very good to hear!