Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
Today I went across a weird case for which I have no explanation, so here I am.
I have two files with identical content, but one is encoded in UTF-8 and the other one is in IBM EBCDIC. Both of them have approximately the same size (336MB and 335MB).
But if I compress the files (ZIP or RAR) one of them is reduced to 26MB (UTF) and the other one to 16MB (EBCDIC).
How can be this possible? The behavior of EBCDIC encoding is always better under compression? Why?
EBCDIC is a format with an 8 bit encoding. The basic EBCDIC character set has plenty of unused space, so not all the potential of the 8 bits is used. But it has codepages for handling non-English languages, such as EBCDIC 284 for latin America.
UTF-8 on the other hand is a multibyte encoding. The English character set is encoded on 7 useful butes. Other unicode characters could be encoded in up to 4 bytes.
The first thought that comes to mind, would be that the UTF8 uses multibyte and this could explain the difference in compression.
The conversion between EBCDIC codepages and UNICODE code map to the 256 first characters. This means that ecoding an EBCDIC file with a local code page, would use a couple of multibyte characters in UTF-8, but each mutlibyte would be 2 bytes long.
Your data shows that UTF-8 encoding is 236MB and EBCDIC encoding is 235MB. This means that at most 1M characters use a multibyte encoding in UTF8 (exactly 1MB if only 2 bytes, less if more bytes are needed). And 234MB use an English character set. So the huge difference in compression (60%) cannot be explained by just because of some multibyte chars.
Furthermore, compression algorithms tend to statistically find out and eliminate redundancies. They should compress the data to the minimum number of bits. The useful information in both files being the same, independently of the encoding, the difference should be minimal.
By deduction, the different encoding cannot explain a huge difference in compression, if the optimal algorithm was used.
According to wikipedia, ZIP files may use the following compression schemes:
Store (no compression), Shrink, Reduce (levels 1-4), Implode, Deflate, Deflate64, bzip2, LZMA (EFS), WavPack, and PPMd
For performance reason, ZIP does not try all of these compressions to determine the absolutely best. It uses some heuristics to chose the most suitable algorithm. The very different statistical distribution of EBCDIC and UTF8 could explain that the heuristic choses different algorithms.
So, with the given information, it seems that ZIP uses a suboptimal encoding scheme, and not necessarily the same for both files.
But this assumption needs an experimental verification. So please check the compression algorithm used in the ZIP files (as explained here on SO). If it's different you have the reason. If it's the same, please update your question and tell us which one so that we can re-investigate with more relevant information.
There is some extraordinary claim. One file of 336MB and one file of 335MB, containing the same data, one compressed to 26MB and one compressed to 16MB.
That's a compression factor of 22 in one case. So either your compression software is very much broken and throws away 75% of the input, or you are compressing some very unusual text.
I wouldn't bother making any guesses about what is going on here until I'm told what exactly the contents of these files is.
