2

I made a test today, how good WinRAR can compress a folder with several times the same picture in it. For that I just put one picture with 300 kB into a folder and copied it there 11 times, so that I had 12 times the same picture, only with different names. I expacted WinRAR to realise this fact and to get a compressed archive with less then the 300 kB of on picture. But the compressed folder was not by much smaller than the normal folder, namely 2.9 MB instead of 3.6 MB. The way I thought WinRAR would work, is by analysing returning patterns in the data and using them as an entry in its "dictionary". By that it would just have to save the picture once in this dictionary and then refer to this entry for each picture, only with a different header (e.g. for the filename).

Why does it not work like that? I mean, there can't be significant differences in the multiple files?

  • 1
    As I understand it, compression algorithms like this work on a file-by-file basis. Even if it treats all of the files as a single data stream, it would still have to know that a large chunk of data is similar or identical to another chunk that is a non-trivial distance away, and I'm absolutely sure that RAR is not smart enough to do that. – Robert Harvey Aug 16 '17 at 15:33
  • Archiving and compression are conceptually unrelated. RAR and ZIP archives contain compressed files. The advantage: individual files can be extracted or replaced cheaply. Other formats like TAR only contain uncompressed files, but you can compress the whole archive (.tar.gz file). Now you can't extract a single file without decompressing the whole archive, but Tar archives can't really do that anyway. – amon Aug 16 '17 at 15:50
  • The basic principle is to look around, find often repeated patterns and assign the shortest codes to the patterns that are most often encountered. As you look for larger patterns and you widen your scope (look further away), you will need more time and memory to do so. And the correlation will be more like exponentially than linearly while gains will likely diminish quickly. So it is just not worth it to compare entire files, which would mean a number-of-files-square performance hit. You have to draw the line somewhere and the file boundary is both effective and convenient. – Martin Maat Aug 16 '17 at 17:24
4

There may not be any differences in the duplicate files, but it takes a lot of effort to notice and verify this. Essentially, you'd have to compare each pair of duplicate files byte for byte in addition to all the lower-level pattern recognition you're already doing. Presumably this is why the WinZIP programmers didn't check for duplicates of entire files.

Speed of encoding is one of the most important attributes of a compressor, and people don't often create archives with many exact duplicates at file-level. Therefore it makes sense to omit a feature that brings great savings only rarely, and then at great cost.

  • But why would compressing two files together be any more complicated than compressing two files individually? The compression algorithm doesn't care what bytestream we feed into it. – amon Aug 16 '17 at 15:54
  • @amon I don't know the details for WinRAR but most people don't want/need to compress everything as a single stream so I doubt WinRAR defaults to it. Compressing everything together has the potential for better compression, but you lose random access - you have to decompress all the files to extract any file. – Doval Aug 16 '17 at 17:18
  • Ahh, ok. I did not know that the algorithm is only looking for patterns in small data parts. I get, that it would require a lot more calculation power to analyse the whole data, but I don't get why I wouldn't be able to extract single files from the compressed archive. What would make it impossible to extract a single file? – jusaca Aug 17 '17 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.