Moving a multi-GB SVN repo to Git

Question

Currently my company has a Visual Studio solition in an SVN repo that's organized as follows:

SolutionFolder (~3.5 GB)
|-> SolutionName.sln
|-> .. Some source code folders... (~250 MB)
|-> ThirdParty (~3 GB)
|-> Tools
    | -> Tool1
    | -> Tool2

Tool1 and Tool2 are build independently (have their own solutions), but produce executables that are used in the main build. ThirdParty folder contains all dependencies for the project, including some pre-compiled 100+ MB .lib files and large libraries like boost.

It's convenient to have it all in one SVN repo so that (1) developer has to do only one check-out, and (2) we don't need to keep track which versions of dependencies we need for each version of the build. On the flip side, it takes a while to check out this repo.

What would be the best way to move this project structure to git? Presumably it's best to exclude ThirdParty and possibly Tools from the main repo, but we'd like to keep ThirdParty easily downloadable in one step, and we like it versioned (and version mismatches between main repo and ThirdParty/Tools would be bad).

At this point I'm not interested in preserving history, just in figuring out how to organize such project.

Answer 1

Use the proper tool for the job. In Windows, that means

Use NuGet for third-party dependencies

That way, you keep the third-party dependencies in a versioned way, but you will not bloat your repository with unneeded stuff. Checkouts are much faster, and the project is organized as it should be. You can enable an option in Visual Studio so it always downloads all dependencies automatically.

Of course you can use a solution that just uses git (another repo, submodules etc), but that's just hacks. Doing it the right way will pay off quickly, and leave you with a future-proof system.

Edit after comments: The best way to use NuGet is to set up a local NuGet source, either on a shared drive, or a full nuget server. Setup shouldn't take more than a few minutes either way. That way, you can guarantee that all the packages you need are always available, no matter where they originated.

Answer 2

You can use submodules for the tools. That way you can keep them in a subdirectory like you do now, and use a separate repo for versioning them. That also means you could clone(checkout) the tools and develop them separately, and that other projects could rely on those repos - and on specific, udateable versions of them too.

You could also use submodules for the third party libraries, but if at all possible I would recommend using a dependency manager for those.

Answer 3

The entities that you turn into git repositories are necessarily the entities that you version and branch; if SolutionFolder/Tools/Tool1 corresponds to one such thing, that's the level of entity. This is because git regards the entire state of the directory tree to be the versionable entity, whereas with svn it is possible (even if not a good idea) to have a trunk, branches and tags anywhere within the tree.

Derived artefacts should not be kept in the repository, nor should external libraries. There are better ways to handle those. (If you're working with Java, consider using a private Maven repository; they're comparatively easy to work with, and integrate nicely with many other things.)

If you're used to a workflow that has everything in one repo for ease of checkout, consider having a script that sets things up instead.

Answer 4

To be honest I wouldn't change anything in your setup. It is exactly what we are doing now. I was playing around with setting up an separate git repository to handle the thirdparty lib that we use but I don't think it weighs up to the cost in portability. Now any developer can just checkout and get started without having to do any manual setup steps. And I any build server/slave can build the project. Unless you have multi repos sharing the thridparty tools I would just stick with your current setup.

What I did play around with was setting up the thirdparty tools in a separate repo. Then I had one simple batch script read a text file with an sha1 ref and checkout the correct version. This would allow me to have different thirdparty versions for different projects. I got this idea from the Facebook Buck build tool. But in the end many developers don't like to use command line tools (MS VC shop here) so I gave up on the idea.

One major reason why not to download your thirdparty libs when you require them (using NuGet) is that if you need to support your product for a long time. In my industry, we need to sometime provide updates for old versions that rely on old thirdparty libs. We do not want to spend lots of time sorting out which libs we can upgrade or not and just use the libs as used in that version. Now imagine you use NuGet, oops... the latest version of the lib you require is 3.98 but you need 2.04..... how to explain to your boss that you need to spend 2 months to upgrade the old version to be able to use the latest libs when he was expecting a small change!