Converting a very large remote subversion repository to a slimmed down Git repository

Question

I have the pleasure of taking over 14 year old subversion repository that consists of two key elements:

1. 111,000 revisions, about 10% of which are substantial;
2. The repository dump is about 73 GB, due to a large number of binary data files that have been updated, added and modified over the years.

Here's what I'd like to do, but I'm not sure if it's possible: I'd like to strip the history of the binary files and only keeping the code changes. Then convert that to git. What are your recommendations?

Answer 1

not sure if it's possible: I'd like to strip the history of the binary files, only keeping the code changes. Then convert that to git.

I did not try this not by myself, but I am pretty sure this is possible when you approach the task exactly in the order you described above:

1. strip the history of the binary files, only keeping the code changes, in SVN first

2. migrate to Git afterwards.

Step 1 can be accomplished by moving the binary files with their history into some temporary folder (inside the repo, for example with svn move). Then you create a fresh copy of them in your projects's local working directory and check them in as if they were new files - so those new files have no history. Then you use the procedure described in this server fault question to get rid of the temporary folder (utilizing svnadmin dump, svndumpfilter, svnadmin load), which deletes the full history.

Note this way whenever you will check out an older revision of the project, the binary files will be missing completely. To avoid this becoming a problem, consider the strategy of keeping the old SVN repo online, as suggested by @RobertHarvey.

Answer 2

OP here. For posterity, I wanted to add what my ultimate solution to this problem was. I will keep the best answer checked, because it's actually the best answer when things go smoothly.

First, I tried the selected answer from Doc Brown: https://softwareengineering.stackexchange.com/a/370419/291998

But the svnadmin dump command failed about halfway into the dump (about 1 day into the dump), with a corrupted revision. This was a consistent point of failure. Attempts to bypass this specific reference, through careful use of the revision flags to svnadmin dump, I was able to produce a full dump, however, attempts to filter binary files out of the 73 GB dump file were met with additional frustration. I did manage to make some dumps of particular branches, but this was useless to me for a full migration.

Ultimately, I ended up using git-svn to do this: nohup git svn init https://myurl.com/projects/myproject/ --no-minimize-url --no-metadata --stdlayout &

Using git-svn on such a large repository was not without issues. It regularly choked during the processing. Fortunately, git svn fetch could be issued to resume to conversion at each stopping point. I wrote a small wrapper script that would continue issuing that command upon failure.

Having said all of this, git-svn is not recommended for this task, see this article: http://esr.ibiblio.org/?p=6778

But this was ultimately the only tool I could get to work. For stripping the binary files, I used BFG repo cleaner https://rtyley.github.io/bfg-repo-cleaner/