4

We store zipped files in the storage of a cloud provider which contain certain fields (metadata). These files are derived from other, larger files. Every time we (re)generate these files, their 'last changed' date is set to the generation time, while the content of the file is identical. When we recreate one of these files, which have previously been stored in the online storage, their file hashes (md5/sha) differ. The reason for that is that the zip algorithm seems to include the 'last changed' information in the .zip file.

We now have cases where we would like to determine if the data stored in the cloud is identical to the newly derived file. A simple comparison of the local hash against the one provided by the cloud provider api fails, for the reasons just stated. The primitive approach would be to download the stored zip file again, unzip, rehash and then do the comparison. This costs money.

A workaround seems to be to artificially force the timestamp on the zip entries to zero. (In Java this is done by zipEntry.setTime(0)). See e.g.: stack overflow This lets us generate .zip files with reproducible hashes, but we loose the information of when the file was generated. This might be a viable workaround, but feels hacky and wrong.

Now, what would be the elegant way to deal with a scenario like this? Is there some smarter hash function of zip files which ignores the timestamps? Is there some better compression algorithm which is reproducible out of the box?

2 Answers 2

2

Proponents of reproducible builds have put some thought into it and came out with the same ideas. The technique to set time may vary. There are even special tools like libfaketime. However, all of them are awkward and hackish.

I believe nothing elegant is invented yet.

1
  • It seems we are going with forcing the timestamps to zero within the archives. Not pretty, but does the job. Thank you.
    – MPIchael
    Commented May 29 at 6:06
2

A few possibilities:

  • store the metadata separately; have for each big file a "foo.shasum" file storing its hash. These can then be handled separately.

  • use the ZIP CRC32; but this has fewer bits and is less collision resistant. It's stored in the zip header. It may be possible to sub-request only the relevant part of the zip file and parse it out.

  • keep the files in git. The Git packfile is a content-addressable store that keeps track of objects by their hash, compresses them, and can tell you which of them has changed.

1
  • We used to have .sig files which stored the hashes, but you inevitably run in to similar problems. Having the files in git does not seem viable as the data is decentral at customer locations and their particular cloud providers. Thanks for the reply!
    – MPIchael
    Commented May 29 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.