2

I made a test today, how good WinRAR can compress a folder with several times the same picture in it. For that I just put one picture with 300 kB into a folder and copied it there 11 times, so that I had 12 times the same picture, only with different names. I expacted WinRAR to realise this fact and to get a compressed archive with less then the 300 kB of on picture. But the compressed folder was not by much smaller than the normal folder, namely 2.9 MB instead of 3.6 MB. The way I thought WinRAR would work, is by analysing returning patterns in the data and using them as an entry in its "dictionary". By that it would just have to save the picture once in this dictionary and then refer to this entry for each picture, only with a different header (e.g. for the filename).

Why does it not work like that? I mean, there can't be significant differences in the multiple files?

Improve this question
3
  • 1
    As I understand it, compression algorithms like this work on a file-by-file basis. Even if it treats all of the files as a single data stream, it would still have to know that a large chunk of data is similar or identical to another chunk that is a non-trivial distance away, and I'm absolutely sure that RAR is not smart enough to do that.
    – Robert Harvey
    Commented Aug 16, 2017 at 15:33
  • Archiving and compression are conceptually unrelated. RAR and ZIP archives contain compressed files. The advantage: individual files can be extracted or replaced cheaply. Other formats like TAR only contain uncompressed files, but you can compress the whole archive (.tar.gz file). Now you can't extract a single file without decompressing the whole archive, but Tar archives can't really do that anyway.
    – amon
    Commented Aug 16, 2017 at 15:50
  • The basic principle is to look around, find often repeated patterns and assign the shortest codes to the patterns that are most often encountered. As you look for larger patterns and you widen your scope (look further away), you will need more time and memory to do so. And the correlation will be more like exponentially than linearly while gains will likely diminish quickly. So it is just not worth it to compare entire files, which would mean a number-of-files-square performance hit. You have to draw the line somewhere and the file boundary is both effective and convenient.
    – Martin Maat
    Commented Aug 16, 2017 at 17:24

1 Answer 1

Reset to default
4

There may not be any differences in the duplicate files, but it takes a lot of effort to notice and verify this. Essentially, you'd have to compare each pair of duplicate files byte for byte in addition to all the lower-level pattern recognition you're already doing. Presumably this is why the WinZIP programmers didn't check for duplicates of entire files.

Speed of encoding is one of the most important attributes of a compressor, and people don't often create archives with many exact duplicates at file-level. Therefore it makes sense to omit a feature that brings great savings only rarely, and then at great cost.

Improve this answer
3
  • But why would compressing two files together be any more complicated than compressing two files individually? The compression algorithm doesn't care what bytestream we feed into it.
    – amon
    Commented Aug 16, 2017 at 15:54
  • @amon I don't know the details for WinRAR but most people don't want/need to compress everything as a single stream so I doubt WinRAR defaults to it. Compressing everything together has the potential for better compression, but you lose random access - you have to decompress all the files to extract any file.
    – Doval
    Commented Aug 16, 2017 at 17:18
  • Ahh, ok. I did not know that the algorithm is only looking for patterns in small data parts. I get, that it would require a lot more calculation power to analyse the whole data, but I don't get why I wouldn't be able to extract single files from the compressed archive. What would make it impossible to extract a single file?
    – jusaca
    Commented Aug 17, 2017 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.