4

This question was inspired by MessagePack, but I'm looking for a general answer about the advantages of in-app vs. external compression.

For network I/O, doesn't the transport protocol (at least optionally) provide some sort of compression? If so, what makes in-app compression better?

For file storage, what makes in-app compression better than external compression (zip, etc.)?

My guess is that in-app compression has more contextual information about what is being compressed, and therefore might perform better in terms of speed and/or compression factor. Obviously my understanding is a little vague. Is there more to it than that?

  • 1
    One problem with compressing everything is CRIME. If you do it manually you can except secrets from compression. – CodesInChaos May 26 '17 at 19:25
  • @CodesInChaos How is that different from excepting them from being output at all? – Kevin Krumwiede May 26 '17 at 19:32
  • Compressing attacker controlled data and secret data opens up a chosen plaintext attack, even if the attacker only sees how much data you're transferring, not the data itself. Google the CRIME attack against TLS for an example. – CodesInChaos May 26 '17 at 20:48
5

In an ideal system (read: well programmed), specific is more efficient than generic, but generic is more broadly applicable. You save development time using generic, you save user time using specific.

A good example would be images. If you used TCP's gzip compression on a bitmap, which has no built in compression, you are applying a purely generic solution. It will look for patterns that it can replicate in less data. For example, 1000 pixels of pure white could be changed from 3000 0xFF bytes into a few bytes, such as an instruction of "replicate bytes" followed by the count "3000" and the value "0xFF". (gzip is way more complicated than that, but you get the gist) But it won't notice that a 100x1000 rectangle of white could be replaced with a single instruction; it'll need 100 instructions, each time it reaches that 3000 block of 0xFF in the stream.

On the other hand, if you use a JPG, which has built in compression, it knows it's an image. It can recognize those regular blocks, it knows that it can skimp in certain areas and the human eye won't notice the difference. If a few pixels are slightly off color, people won't notice, but the machine can fit it into a pattern that makes it more compressed.

But if you tried to apply that compression to binary data, it'll corrupt it.

Alternatively, if you used a lossless image compression type, like say, PNG, it'll have all of that image theory behind it to recognize the ways that it can reduce image size. While you could apply this to regular data without corruption, it would be significantly less useful due to the patterns it's looking for being all wrong for the patterns in the data.

Gzipping on top of that can also find patterns in the patterns to make it even smaller.

The main advantage that MessagePack seems to have over gzip (from a cursory look) is that it eliminates the closing elements (ie, the closing quote, the closing brackets, the commas), cutting the syntactic overhead of json in half, at the cost of making it less readable and human editable. Gzip can't do that.

It doesn't appear to compress the actual data, which gzip is good at, so you should use the two compressions to complement each other. Gzip in TCP on top of MessagePack in application.

However, there is the possibility (I don't know, you'll have to test it) that gzip could handle the patterns in regular json better than the patterns produced by MessagePack, and so could produce a smaller size. I find that doubtful, though, as gzip is more intended for general binary than text specifically.

  • Good answer. I would add that since compression isn't free, performance-wise, applying application-level logic to avoid wasting time gzipping JPGs to bo effect is also an advantage. – Avner Shahar-Kashtan May 27 '17 at 5:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.