How is processing done when streaming data?

Question

If streams in programming is about reading data in chunks, how then is it possible to process the data in the scenarios where the data cannot be processed bit by bit, but the processing logic needs the full data?

I can imagine in some cases it might be possible to process data bit by bit, but it is not hard to imagine that in other scenario, the shape of the data or the processing algorithm needs to have the full data before the processing can effectively be applied.

How is this scenario taken care of?

Answer 1

how then is it possible to process the data in the scenarios where the data cannot be processed bit by bit, but the processing logic needs the full data?

It's simple: Use pre-processing and meta data. For example, without every chunk, you don't know how many chunks there are, unless something upstream counted them and told you. You don't have to count them if the meta data tells you how many there are.

That may be a trivial example, but formats and even transmission protocols have been devised specifically to solve this exact problem. You can only work with what you have.

When people use the word “streaming” they're usually talking about transmission over the network, but it's the same problem when parsing a file from the hard drive. You can load the entire file into memory (sometimes called slurping or batch processing) or you can load a line/chunk at a time (effectively streaming). Which one is appropriate depends on the size of the file, size of the memory, and your parsing/processing needs.

If you don't mind looking at a few lines of Ruby code, I recommend referring to the comparison of performance metrics in this SO answer. It shows that there are some non-linear impacts to consider. Every coder should keep this stuff in mind before assuming there's only one obvious answer here.

Answer 2

If you truly need the full data before processing then just read it all first before processing it... unless of course you have control of the data being sent, then the answer from u/candied_orange could be a better way to go.

But many scenarios where you think you need the full data can actually be converted to a streaming implementation. The real question to ask is: "What are the pros and cons of processing this in a streaming manner" where the common tradeoff for streaming is usually a lower memory footprint but giving up keeping a re-usable data-structure (for example SAX or StAX vs DOM XML processing). However, streaming processors could always still build those data-structures, so having a streaming implementation available with that ability is usually the best of both worlds.

Answer 3

There is no conflict between streaming on the one hand and processing data on the other hand.

To process any data read from any medium, you will always need to read ahead to some extent. Whether some will be a couple of bytes or the entire contents of a file is irrelevant.

Answer 4

We parse streams into data.  Parsing is a task of recognition according to some ruleset governed by a "language".

When a parser recognizes something, it can transform that into a more program-friendly data structure.  Things it doesn't recognize are either saved and accumulated until recognition can happen given more of the stream (or else, there is an error in parsing — the stream is outside of the language).

In many ways it depends on the "language" being used in the stream as to how much can be processed in pieces before having the whole stream.

Message & File formats are one form of language, there are also many text-based languages, as well as custom application-specific forms.

Message forms can be designed to be more suitable to streaming, by including headers with sizing and other metadata.  HTTP is one such format.  Some image forms provide for initial low resolution content, such that an initial though blurry image can be displayed while the full image is still being received.