24

Our project is about 11GB, 10 of which are binary data (.png images). Consequently, a git diff or git status operations take up more than a minute. Fortunately all data files are separated into a folder with the wonderful name data. The assignment is "Avoid compressing, diffing and other costly operations on binary files."

  • It was considered splitting the project into two repos. Then data would be an external repo, that is checked out by the main source code repo. It was decided that the overhead of keeping the repos in sync would be too much, especially for the artists, who work with the data files.

  • Explicitly telling git those files are binary, excluding files from diffs were considered, but those seem like only a partial solution to the question.

I feel that git attributes is the solution, but how? Or is there a better architecture than a monolithic repo?

Improve this question
6
  • 2
    The first big question here is how important are those data files. Does your program need all of those images available in order to do anything useful, or can it get away with a small subset during typical development/testing?
    – Ixrec
    Mar 11, 2016 at 17:02
  • @Ixrec, the images are actually more important than the source code. All of them must be present, and .png checksums are checked always for corrupt files.
    – Vorac
    Mar 14, 2016 at 9:56
  • 1
    Why isn't this question on stack overflow? The Q. Seems exactly suited to it.
    – spirc
    Mar 17, 2016 at 0:41
  • 1
    @spirc this question straddles the line between "help with a software tool" which is on-topic at SO, and "version control strategy" which is on-topic here. Since it is not asking for what git command to execute to do something, it is not clearly on the SO side of the line so I voted to leave it open here.
    – user22815
    Mar 18, 2016 at 21:49
  • 2
    @spirc Questions on source control fall under the topic of "software configuration management." As long as a source control question is not also about "how to use specific tools" it should be on-topic. If you reread my previous comment I mention that it is not asking about which git command to execute so it is not about software tools: it appears to fall under the SCM category.
    – user22815
    Mar 21, 2016 at 4:41

3 Answers 3

Reset to default
18

You can use git-lfs or similar tools (git-fat, git-annex, etc.). Those tools basically replace the binary files in your repo with small text file with hashes, and store the actual binary data in a non-git way - like a network share.

Makes diffs and everything superfast as only hashes get compared, and is - at least for git-lfs - transparent to the user (after installing once).

Afaik git-lfs is supported by github, gitlab, VisualStudio, and is open source.

Improve this answer
4
  • 2
    Have you tried using git-lfs on a project with many gigabytes of assets with a mixed developer/artist team? I'm interested to know if people are using git-lfs for projects such as games and animation. Since its still fairly new at time of writing. From my own experience the barrier of entry to git for less technical users is already very high, so having an extra layer for file management on-top of it - may be difficult for people to use unless they're already comfortable with git.
    – ideasman42
    Mar 15, 2016 at 3:19
  • Only for up to around ~1GB of data, sorry. But git-lfs should add no additional steps for endusers, it should be completely transparent.
    – kat0r
    Mar 15, 2016 at 8:40
  • This seems to be the correct answer, if some problems arise during the integration I will report back here. So the installation procedure needs to be completed only once on the server, and not on each client machine?
    – Vorac
    Mar 15, 2016 at 12:06
  • Afaik you need to install a small client addin, too, check the github page. But that should be easy to roll out with a group policy/simpler than any alternative.
    – kat0r
    Mar 15, 2016 at 12:12
2

Use both GIT & SVN repos

If the binary files can be separated logically from the source, you might consider using git for text files, and a non DVCS such as subversion for the binary files.

A project I work on does this since we have many GB for per-compiled libraries (for OSX/Win32 dependencies), which we need to keep versioned.

On the other hand if you have non-technical users, using two version control systems may be problematic. However if the artists aren't working on code you could provide a script to perform the update, and they can use subversion to commit binary assets.

Use SVN (with git svn)

While this trade-off isn't always so nice for developers who are used to using regular git, you could use SVN for the main repository, and developers can use git svn tools.

This does make it a little more work for developers using git, but means for everyone who isn't familiar with DVCS (or VCS in general) - they can use SVN's simple model without having to use multiple complex version control systems.

git-lfs is an option too, but I didn't use it so can't speak to how well it works.

Improve this answer
0
0

Specifically for PNG, you are lucky, as you could use png-inflate as a git filter, which "uncompresses" PNGs when checking them into the repo, which might reduce the repo size by quite a bit. Every user has to install this separately, but you could provide a script for that, and they only have to do that once.

I'd recommend to apply it to the whole history of your project, and then see whether it is any smaller, and if the diffs are faster and so on.

side-note: The same thing can be done for ZIP based files (which includes MS Word, Libre Office and FreeCAD, amongst others) with ReZipDoc.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.