3

I watched two YouTube videos where in:
1st one - there are concepts of batch processing and stream ingestion and in the
2nd one - there is a comparison between batch and real time processing.

Is it fair to say that when we refer to streaming and real time processing then we are actually talking about exactly the same concept?

0

4 Answers 4

7

As the other answer mentions, a 'stream' is an ordered sequence of events, and the concept of 'streaming' typically refers to transmission of events.

'Streaming' also generally implies that events are ordered by time and that each event is associated with a fixed point in time. Therefore it is also important for consumers to be able to process those events in a way which is sensitive to their time-ordering and point-in-time origin.

Real-time typically refers to a series of events which must be processed or consumed "as-they-happen" - near-instantaneously or within a reasonable (minimal) lag or delay.

Time-ordering of events is still important, with an additional constraint that the fixed point-in-time needs to be very recent (almost "right now" or within a short timeframe).

In a real-time streaming scenario, events are typically only relevant for a very short timespan. If they are lost or delayed for too long, this can have a negative impact on the consumer (because it didn't receive the information in a timely manner).

Clearly there are some forms of streaming which are only meaningful when consumed 'live', so these could be referred to as real-time streams.

Other forms of streaming, which are not happening in real-time, merely involve replaying a series of events which had been effectively pre-recorded and stored somewhere; the consumer is not expecting to consume in real-time, although still relying upon time-ordered events with fixed times relative to the beginning of the stream.

The latter opens opportunities which are obviously impossible in real-time - such as changing the processing speed, time-jumps and rewind/replay. In some cases the consumer may even want to alter the start/end point, or perhaps playback the events in-reverse.

2
  • I don't agree with the claim that streaming implies any sort of ordering. It often helps or happens naturally but it's not given. You can for example stream game assets into memory to create seamless loading. It's more correct to say that streaming means you're consuming and/or producing something without holding the whole dataset in working memory (or even on the host).
    – Stylpe
    Apr 23 at 17:01
  • @Stylpe The context of the question (based on the YouTube video which the OP linked in the question) is about data pipelines and transmission of events, where 'Streaming' has a particular meaning, specifically relating to events, which is rather different to your example of loading data into memory (The OP is not referring to reading data into a memory buffer, but an event stream such as those used in ETL pipelines). Streamed events in this context must be ordered otherwise it defeats the point of using event streaming in the first place. Apr 23 at 21:51
9

Depending on the context these may mean the same. But to me these concepts are quite different.

  1. Streaming is about communication, not about "real time". Characteristics of a stream would be: it has two endpoints, one endpoint writes, the other reads (while technically streams can be bidirectional, conceptually it doesn't differ much from two streams in opposite directions). It also preserves order of writes, and doesn't loose data. So it is reliable. For example TCP stream.
  2. Real time processing is, well, real time processing. Pretty much: fast processing of data. Often with some deadline constraints, which are hard requirements. Which unfortunately are not well defined, for one person/company it can be say 2s, for the other one 10ms. Fast enough, whatever that means.

Streams don't have to be fast. In particular, the TCP stream tends to be slow when packet loss occurs (in which case it needs to retry). In fact, the price for being reliable is worse performance.

Real time processing does not have to be based on a streams. First of all, it doesn't have to be related to communication at all (e.g. real time computer games). But even in the context of communication, for example real time game networking is often done over unreliable layer like UDP. Because it is faster, while TCP guarantees are not necessary.


That being said, depending on context these can mean similar thing. Especially when compared to batch processing.

Either way, one should be careful about those terms. Always read details about what actually is being compared.

5
  • 6
    Case in point for being careful with the term "real-time": In my work, real-time means that there is a deadline on how fast the software must respond to an event. In some cases, that response time is measured in seconds, but failing to meet the deadline means that the product is considered defective/broken. A hard real-time requirement. Apr 19 at 7:13
  • @BartvanIngenSchenau fair point, I've updated the answer.
    – freakish
    Apr 19 at 7:23
  • 2
  • Probably I was confusing these terms also because I heard many times about Kafka Streams real-time processing..
    – bridgemnc
    Apr 20 at 6:11
  • 1
    I second @BartvanIngenSchenau 's comment. Real-time means whatever that individual thinks it means at that particular time. I've seen realtime mean exactly what you say, I've seen realtime mean "roughly runs as fast as the inputs come in" and anything inbetween.
    – Cort Ammon
    Apr 21 at 4:33
3

At the margin the distinction may blur, but typically a stream is either continuous (in the sense of incessant) or reactive immediately, whereas a batch is scheduled explicitly on a rhythm or involves some kind of accumulation.

The terms are usually used in a context where a distinction in meaning is clear, rather than each term defined in isolation.

3

There are some parts of streaming that { can, must, should } be real-time. Whether this is the case, and if so, which of the requirement words applies, depends on what particularly you're streaming. In relation to streaming, real-time is an attribute.

Real-time systems are generally defined as having a fixed maximum amount of time (or something related) it takes to process a unit of work. A common example is, "at least as fast as the fastest a human can perceive," such as about 70 frames per second for screen updates.

Often, a real-time system will have a policy of what to do with a unit of work that has expired, or passed its time window. It could be some dramatic failure with alarms and stuff or just silently dropping the packet.

Real-time processing, when referring to computer systems, also usually implies non-preemptable. Assuming you can do whatever computational work you need to quickly enough to match the rate at which stuff arrives, you probably can't bear the delay of running something extra like a garbage collector interrupting when it feels like it. In many of these situations, the, "complicated," part of the processing is done in dedicated hardware, which is almost always real-time.

As stated in other answers, streaming requires a connection and therefore all the things that come with that. Real-time processing in the pure sense, does not.

If you're streaming video, many modern computers can do most video processing easily, so real-time isn't required in those cases. There are many other things that are streamed though, and some of them may or may not require real-time processing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.