2

I migrated some files from one Version Control System to another. I have to prove that content are identical (between source and destination, denote 1 , 2). There are a few zips files (lets call them A B C) each of which contains hundreds of files. I am looking at best way to do CRC comparison between content in old VCS against the new one.

1) Generate CRC on each ZIP file as a whole and compare the CRC of 2 corresponding zips file. ZIP-ZIP CRC comparison.

Obviously, this approach will be easier. But I don't know what will include in calculation of CRC. Or even if CRC of two zip files with identical file content might be different.(modified date?).

2) Compare CRC of each files in zip against corresponding files. File-File CRC Comparison

with this approach, I will have to write a script that goes through each file in zip (say A1) and extract their CRC. Build a list with [path file name, crc]. do the same for zip(A2). Compare list.

Have anyone ever done something like this?

8
  • 2
    Those zip files might contain variable data like datum / timestamps.
    – πάντα ῥεῖ
    Commented Apr 9, 2018 at 18:56
  • Yes, this is just like a comparison of the files in two directories.
    – Erik Eidt
    Commented Apr 9, 2018 at 19:07
  • but these are not text file they are in some propriety format.
    – meah1991
    Commented Apr 9, 2018 at 19:10
  • 1
    Skip the CRC and just compare the files normally.
    – whatsisname
    Commented Apr 9, 2018 at 19:11
  • 2
    @Clearer: In order to compare both files, you need to read both files. In order to compare checksums of both files, you need to read both files, then compute the checksums. The amount of work you need to do when using checksums is strictly greater than when just comparing the files. If you pre-compute the checksums, you can amortize this cost among many comparisons, but how often do you compare the exact same two files? In the description of the OP it certainly sounds like one time event.
    – Jörg W Mittag
    Commented Apr 10, 2018 at 13:53

2 Answers 2

Reset to default
5

If you're sure the compression algorithm used to create both zip files is identical then you can just compare the zip files.

Otherwise you will need to decompress the zips and compare contained files.

Hashes generated when compressing could be used to speed up comparisons if you'll accept the chance of collisions causing false positives. This can quickly show files to be different.

But collisions mean the best you can do with a hash is show files to "very likely" be identical. With enough bits and a good hashing algorithm we're talking about odds akin to winning the lottery. In a practical application you'll have to decide if speed is worth the risk.

If you're serious about Proof the files are identical you can't ignore unlikely cases. Quarters sometimes land on their edges. Sometimes hashes collide. But sometimes bits flip on you randomly and go undetected. So don't think a bit by bit comparison of the uncompressed files is guarantied to give you a perfect proof either. What you get is lots of bits giving you really good odds.

This last is when the CRC is useful. Not as a digest. It's an error check. It makes a bit copy error less likely to go unnoticed. Still not perfect because CRC bits can be badly copied as well.

So there just isn't a perfect proof. Do it right and you can have fantastic levels of confidence, if you have time for that.

5
  • if one zip file is a copy of another, would it be reasonable to assume they will have same CRC?
    – meah1991
    Commented Apr 9, 2018 at 19:24
  • 7
    Reasonable but pointless. CRC is for detecting errors in copying. Not for comparing files or creating a digest. See crc vs hash
    – candied_orange
    Commented Apr 9, 2018 at 19:32
  • 1
    Just to be clear: neither CRC nor a hash can prove the files are identical. They can only prove the files are different.
    – Jörg W Mittag
    Commented Apr 10, 2018 at 8:12
  • @JörgWMittag better?
    – candied_orange
    Commented Apr 10, 2018 at 8:35
  • @JörgWMittag true, but even with poor hash algorithms, the chance of getting a hash collision is pretty low. If it hashes to the same and is of the same size, then there is a good chance that it is the same file.
    – Baldrickk
    Commented Apr 11, 2018 at 10:37
0

This should do the job (in bash):

# Generate a file containing MD5 hashes for all files in A.zip
unzip A.zip -d A_dump
(cd A_dump && find . -type f | xargs md5sum > ../A.md5)

# Compare the hashes to the content of B.zip
unzip B.zip -d B_dump
(cd B_dump && md5sum --check ../A.md5)

You can of course use a better hashing algorithm if you want, and use the appropriate tool like sha1sum or sha256sum but MD5 is fast and even though collisions are possible, they are very unlikely unless they are crafted.

For the python version, the zipfile and the hashlib modules provide all you need.

2
  • If you have bash, then you probably have diff, so diff -r A_dump B_dump will tell you not only if they are the same or different, but WHAT is different.
    – wjl
    Commented Apr 21, 2018 at 20:15
  • That's a second steop. We're talking about integrity, so we only care at first if files are different or not, and that's what you can automate and test. Getting the whole diff is useless at that point. The diff is usefull if you want to investigate why they are different, but even that information may be of little value. What you want to know is to which revision on both systems is used, and diff won't give you that.
    – liberforce
    Commented Apr 23, 2018 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.