Two things you can do in Java:

  1. Send a gzipped JSON body in response to an HTTP request
  2. Send a StreamingOutput response to an HTTP request, where you begin sending a response before you know the entirety of the response (e.g. if reading entries from a database)

According to my testing, you can do both at the same time - i.e. gzip a stream of data before you know the entirety of the data.

How is it possible to gzip an incomplete stream? My limited understanding of compression is that it replaces, for example, the most common word with a single byte instead. But figuring out which word is "most common" requires knowing the entire data set. Right?

  • 3
    The stream is sent back in chunks -- those chunks are gzipped. Jun 26, 2019 at 4:58
  • 3
    As you already noticed, your understanding is limited. Many compression algorithms consider the data already seen only, not the future data. In theory, they could employ a fixed size buffer to see ahead a bit, but the deflate algorithm that's the core of gzip does not work that way. Jun 26, 2019 at 5:01

1 Answer 1


Your understanding concerns static compression, where the whole dataset is available when you begin compression. In a very simplified way, yes it replaces common sequences with smaller tokens to achieve compression. There are many cases where this won't even result in efficient compression (if done in a naive way), such as skewed data distribution (think "AAABBBCCC" vs. "AAAAAAABC").

More interesting is dynamic compression, which doesn't require before-hand knowledge of the full dataset. For the common CS homework algorithm, see Adaptive Huffman Coding (compared to regular Huffman Coding ) which builds the coding table as it processes the data. In addition to enabling compression of a stream, it also allows the coding table to encode data efficiently in different parts of the stream, regardless of previous data (think "AAABBBCCC").

The dynamic building of the coding table however does establish a dependency on the previous data. To correctly decode the last byte, you need to decode the first and all the following bytes.

In an unreliable environment such as the network, you don't want to resend the whole dataset if a single byte is corrupt (which would break the decoding), so there are lots of other things involved, such as chunking where you divide the payload into parts and compress them independently. A wrong checksum of the chunk would mean only that chunk is resent.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.