I've read on several discussions here and on SO that DVCS repositories use about the same or less space than their centralised counter-parts. I may have missed it, but I haven't found a good explanation of why that is. Anyone know?
6
-
5Are the followinfg posts the one you read? stackoverflow.com/questions/7727791/… or stackoverflow.com/questions/8657710/… or stackoverflow.com/questions/456336/…– VonCCommented May 13, 2012 at 13:49
-
1I haven't, thank you! So I understand from those that there are two answers: compression using zlib and saving objects as packfiles when possible. The examples from Mozilla is also great!– Alex FlorescuCommented May 13, 2012 at 14:40
-
1@Alex No, that misses the main reason. SVN saves complete snapshots, Git and Mercurial only save the HEAD revision and diffs. Using conventional compression can give you best-case rates of about 60–80% compression. Using diffs could give you as much as 99%. These numbers are pulled out of my ass though – real numbers might differ; the tendency will be the same though.– Konrad RudolphCommented May 13, 2012 at 19:16
-
@KonradRudolph, isn't that what packfiles are all about?– Alex FlorescuCommented May 14, 2012 at 12:11
-
@Alex Not really. As far as I know the packfile is additionally packing multiple files into one. This isn’t necessarily related.– Konrad RudolphCommented May 14, 2012 at 12:30
|
Show 1 more comment
1 Answer
From my own experience, the following statements are all true:
- Git is very efficient in storing text files, and only storing these files that were changed. so when doing a comparison of SVN and Git to compare the repository sizes, they may be similar, or there may be even a small advantage for Git.
- This is completely wrong if you compare the size of repositories where a significant amount of files are office files (like MS word, excel, powerpoint, ...). Here Git stores complete copies as well, which means that 10 small changes on a powerpoint slide stack results in 10 complete copies, where Subversion only stores a binary diff, which may be a factor of 100 smaller.
If you compare the checkout location (which is a repository in itself with Git), the story is totally different:
- Subversion stores for each file a complete copy, so the size of your checkout location is normally 2 times the size of the files themselves.
- Git stores the complete history of the repository locally, so depending on the size of the history, this may be smaller or much bigger than that of the Subversion checkout copy.
If you compare the amount of bytes you have to down- or upload, it is different again.
- Subversion has normally to send or receive less bytes, because it only sends differences. It has to do that on every commit and update.
- Git has to get the whole repository (initially), and then sends complete files (compressed?) which is not such different for text files, but may be different for binary files. And yes, Git only does that when you push or pull something to the remote repository.
So at the end, you compare apples with oranges, and depending on what you want to do with Subversion or Git, the result may be different.
@jk asked about complete copies or binary diffs, and I could not answer that question. I asked Matthew McCullough which gave a Git workshop lately at Jax 2012 (which I visited). He has taken the time (thank's a lot to him) to explain with a detailed gist the inner working of Git. So yes, there is a compression working there (and I will do an experiment with a microsoft office file as well and will compare that with his gist), but no, the compression is done on the whole file. Citing from his gist:
Loose objects are written in compressed, but non-delta format at the time of each commit.
-
1are you sure that git stores complete copies of office files? I think it also stores binary diffs. of course the really issue with these sort of files is they are often already compressed so a small change can cause the whole file to change– jk.Commented May 14, 2012 at 11:46
-
2Asked someone (by email) which knows much more than me, and will include his answer in my answer then.– mliebeltCommented May 14, 2012 at 12:21
-
6Git treats text and binary files exactly the same way in all and any regards with respect to storage. The loose vs. packed objects is unrelated to text vs. binary. The reason binary files often result in much larger diffs than text files is that many binary formats (including all the new office formats) are already compressed and thus even a small change in content often causes big change in the resulting binary blob. This is equally of concern to git and subversion, but subversion only takes the penalty on server, while git everywhere. Commented May 15, 2012 at 7:36
-
4The loose vs. packed objects has nothing to do with text vs. binary. It is amortization of the difficult work of finding the binary diffs. Speed is important feature of git, so during regular operation, git only zips the new data and slaps them in the repository. This is loose objects. Than when you ask it to by calling
git gcor too many loose objects accumulate, it finds good candidates to delta-compress them against (git can diff against other than previous version), stores the deltas in a "pack" and removes the loose objects. Commented May 15, 2012 at 7:45 -
3