What to do about large svn history when moving to git?

Question

Edit: unlike some similar questions such as Moving a multi-GB SVN repo to Git or Managing large binary files with Git My scenario doesn't involve several subprojects that can be easily converted into git submodules, nor a few very large binary files that are well suited for git-annex. It is a single repository where the binaries are the test suite which is tightly coupled to the main source code of the same revision, much like if they were compile time assets such as graphics.

I'm investigating switching an old medium/large sized (50 users, 60k revisions, 80Gb history, 2Gb working copy) code repository from svn. As the number of users have grown, there is a lot of churn in trunk, and features are often spread out on multiple commits making code review hard to do. Also without branching there is no way to "gate" bad code out, reviews can only be done after it is committed to trunk. I'm investigating alternatives. I was hoping we could move to git, but I'm having some problems.

The problem with the current repo as far as git goes is size. There is a lot of old cruft in there, and cleaning it with --filter-branch when converting to git can cut it down in size by an order of magnitude, to around 5-10GB. This is still too big. The biggest reason for the large repository size is that there are a lot of binary documents being inputs to tests. These files vary between .5mb and 30mb, and there are hundreds. They also have quite a lot of changes. I have looked at submodules, git-annex etc, but having the tests in a submodule feels wrong, as does having annex for many files for which you want full history.

So the distributed nature of git is really what's blocking me from adopting it. I don't really care about distributed, I just want the cheap branching and powerful merging features. Like I assume 99.9% of git users do, we will use a blessed, bare central repository.

I'm not sure I understand why each user has to have a full local history when using git? If the workflow isn't decentralized, what is that data doing on the users' disks? I know that in recent versions of git you can use a shallow clone with only recent history. My question is: is it viable to do this as the standard mode of operation for an entire team? Can git be configured to always be shallow so you can have a full history only centrally, but users by default only have 1000 revs of history? The option to that of course would be to just convert 1000 revs to git, and keep the svn repo for archeology. In that scenario however, we'd encounter the same problem again after the next several thousand revisions to the test documents.

• What is a good best practice for using git with large repos containing many binary files that you do want history for? Most best practices and tutorials seem to avoid this case. They solve the problem of few huge binaries, or propose dropping the binaries entirely.
• Is shallow cloning usable as a normal mode of operation or is it a "hack"?
• Could submodules be used for code where you have a tight dependency between the main source revision and the submodule revision (such as in compile time binary dependencies, or a unit test suite)?
• How big is "too big" for a git repository (on premises)? Should we avoid switching if we can get it down to 4GB? 2GB?

Answer 1

Wow, that's a long question (and a complex problem). I'll try to have a go at it.

I'm not sure I understand why each user has to have a full local history when using git?

This is a central design decision with git. For the exact reasons you'd need to ask the author (Linus Torvalds), but as far as I know, the main reason is speed: Having everything local (on a fast disk or even cached in RAM) makes operations on history much faster by avoiding network access.

The biggest reason for the large repository size is that there are a lot of binary documents being inputs to tests. These files vary between .5mb and 30mb, and there are hundreds. They also have quite a lot of changes.

That is the point I would think about first. Having so many constantly changing binary files in source control seems problematic to me (even with SVN). Can't you use a different approach? Ideas:

• Unlike source code, a 3 MB binary file is probably not written by hand. If some tool/process generates it, consider integrating that into your build, instead of storing the data.

• If that is not practical, binary files are typically better off in an artifact repository (such as Artifactory for Maven & co.). Maybe that is an option for you.

I have looked at submodules, git-annex etc, but having the tests in a submodule feels wrong, as does having annex for many files for which you want full history.

Actually, this looks like git-annex would fit perfectly. git-annex basically allows you to store file contents outside a git repository (the repository contains a placeholder instead). You can store the file contents in a variety of ways (central git repo, shared drive, cloud storage...), and you can control which contents you want to have locally.

Did you maybe misunderstand how git-annex works? git-annex does store full history for all the files it manages - it just lets you choose which file contents you want to have locally.

Finally, about your questions:

What is a good best practice for using git with large repos containing many binary files that you do want history for?

In my experience, the options usually are:

• avoid the need for binaries in the repo (generate them on demand, store them elsewhere)
• use git-annex (or a similar solution, such as Git LFS)
• live with a big repo (not all git operations are affected by big files, and if you have a fast computer and drive, it can be quite workable)

Is shallow cloning usable as a normal mode of operation or is it a "hack"?

That might be doable; however, I don't think this will solve your problem:

• you'd lose of git's benefits that come from having full history, such as quick searching of the history
• merges can become tricky, because AKAIK you must have at least the history back to the branch point to merge
• users would need to re-clone periodically to keep the size of their clone small
• it's just an uncommon way of using git, so you'd likely run into problems with many tools

How big is "too big" for a git repository (on premises)? Should we avoid switching if we can get it down to 4GB? 2GB?

That depends on the structure of the repo (few/many files etc.), on what you want to do, on how beefy your computers are, and on your patience :-).

To give you a quick idea: On my (newish, but low-spec) laptop, committing a 500 MB file takes 30-60s. Just listing history (git log etc.) is not affected by big files; things like "git log -S" which must scan file content are very slow - however, the speed is mainly dominated by I/O, so it's not really git's fault.

On a 3 GB repo with a handful of revisions, "git log -S" takes about a minute.

So I'd say a couple of GB is ok, though not ideal. More than 10-20 GB is probably pushing it, but it might be doable - you'd have to try it.

Answer 2

As the number of users have grown, there is a lot of churn in trunk, and features are often spread out on multiple commits making code review hard to do. Also without branching there is no way to "gate" bad code out, reviews can only be done after it is committed to trunk

Moving to git will not solve these problems, they're issues in how you use the tool and if you use git the same way, the issues will remain.

You can branch in svn just as easily in git, and merging is generally just as easy and has the same pitfalls. Git was designed for working with the kernel source code, so it made some assumptions that may not apply in all cases, such as yours with big binaries and massive histories. The intention behind a DVCS is that every user effectively works alone and only collaborates afterwards - ie they have their own repo (a copy), work how they like and then push the changes to anyone else who wants it. A federated system used in linux kernel development is perfect for this - you push your changes to the next guy up the chain who merges it with his codebase and then pushes it to the next guy until it gets to Linus who puts it in the release. Most teams use git similarly, but with only 1 upstream guy which is often a server-side 'gold' repo, somewhat making git similar to a disconnected CVCS.

So I'd look at changing your workflow first, only migrating to git once you have a better way of working. Implement branching and merging in SVN, if you don't rename files or directories merging goes quite well.

Answer 3

Look into GCC mailing list. Migrating the GCC compiler's source tree from SVN to GIT is discussed right now (august & september 2015), while keeping GCC's history. See e.g. repository for the conversion machinery & Acceptance criteria for the git conversion mail threads; you'll find references to tools and procedures related to the conversion (which is not as straightforward as it seems; the conversion of such a large code base history needs 36 hours and about 64Gbytes of RAM, IIRC)

Answer 4

If converting the whole SVN repository into Git results in huge repository which is unfeasible to clone, you can try using SubGit for creating smaller Git mirrors for certain parts of your Subversion repository.

For instance, you can import and sync some subdirectory of your SVN repository http://domain/repos/trunk/project/src:

subgit configure --layout auto --trunk trunk/project/src http://domain/repos project.git
edit project.git/subgit/config
edit project.git/subgit/authors.txt
subgit install project.git

For more details on using SubGit refer to its documentation.

As soon as you have Git mirror of that directory, you can use Git repository for submitting new changes that immediately get reflected in SVN repository. Since you only sync certain part of SVN repository that shrinks the size of converted Git repository significantly and you still can create branches, merge them, employ any workflow from Git side.

Alternatively, you can import the whole SVN repository but exclude large files from the sync:

subgit configure --layout auto --trunk trunk http://domain/repos project.git
edit project.git/subgit/config
...
[svn]
    excludePath = *.bin
    excludePath = *.iso
...
edit project.git/subgit/authors.txt
subgit install project.git

Resulted Git repository should have reasonable size and developers can still use Git for submitting their changes to Subversion repository.

Note that this solution should work well for you if you're ready to keep Subversion server running and use Git alongside your SVN repository.

Disclaimer: I'm one of SubGit developers; SubGit is commercial software with a number of free options available.

Answer 5

I wold approach your situation in the following way:

1) Initialize a git repository in the same directory as your SVN repo. Do git init and git remote add origin to start that git repo. That way you can continue to commit on SVN and git separately without dealing with a full conversion from one to the other until you are ready.

2) Actively use bfg and filter-branch tools to try and shrink your git repo, as discussed here: https://confluence.atlassian.com/bitbucket/reduce-repository-size-321848262.html

3) Use git-annex, or Git LFS, or just an external storage server for your large binaries (transporting the files using shell scripts at build time).

4) Once you are comfortable with the merging/branching strategy in your git repo, and are comfortable with the size of your git repo, you can then do a full migration from your svn to git.

Hope this helps.