11

According to the Kafka site:

"Kakfa is used for building real-time data pipelines and streaming apps."

Searching the internet far and wide, I've found the following generally-accepted definition of what "stream data" is:

  • Stream data is data that flows contiguously from a source to a destination over a network; and
  • Stream data is not atomic in nature, meaning any part of a flowing stream of data is meaningful and processable, as opposed to a file whose bytes don't mean anything unless you have all of them; and
  • Stream data can be started/stopped at any time; and
  • Consumers can attach and detach from a stream of data at will, and process just the parts of it that they want

Now then, if anything I've said above is incorrect, incomplete or totally wrong, please begin by correcting me! Assuming I'm more or less on track, then...

Now that I understand what "streaming data" is, then I understand what Kafka and Kinesis mean when they bill themselves as processing/brokering middleware for applications with streaming data. But it has piqued my interests: can/should "stream middleware" like Kafka or Kinesis be used for non-streaming data, like traditional message brokers? And vice versa: can/should traditional MQs like RabbitMQ, ActiveMQ, Apollo, etc. be used for streaming data?

Let's take an example where an application will be sending its backend constant barrage of JSON messages that need to be processed, and the processing is fairly complex (validation, transforms on the data, filtering, aggregations, etc.):

  • Case #1: The messages are each frames of a movie; that is one JSON messgage per video frame containing the frame data and some supporting metadata
  • Case #2: The messages are time-series data, perhaps someone's heartbeat as a function of time. So Message #1 is sent representing my heartbeat at t=1, Message #2 contains my heartbeat at t=2, etc.
  • Case #3: The data is completely disparate and non-related by time or as part of any "data stream". Perhaps audit/security events that get fired as hundreds of users navigate the application clicking buttons and taking actions

Based on how Kafka/Kinesis are billed and on my understanding of what "streaming data" is, they seem to be obvious candidates for Cases #1 (contiguous video data) and #2 (contiguous time-series data). However I don't see any reason why a traditional message broker like RabbitMQ couldn't efficiently handle both these inputs as well.

And with Case #3, we're only provided with an event that has occurred and we need to process a reaction to that event. So to me this speaks to needing a traditional broker like RabbitMQ. But there's also no reason why you couldn't have Kafka or Kinesis handle the processing of event data either.

So basically, I'm looking to establish a rubric that says: I have X data with Y characteristics. I should use a stream processor like Kafka/Kinesis to handle it. Or, conversely, one that helps me determine: I have W data with Z characteristics. I should use a traditional message broker to handle it.

So I ask: What factors about the data (or otherwise) help steer the decision between stream processor or message broker, since both can handle streaming data, and both can handle (non-streaming) message data?

5

Kafka deals in ordered logs of atomic messages. You can view it sort of like the pub/sub mode of message brokers, but with strict ordering and the ability to replay or seek around the stream of messages at any point in the past that's still being retained on disk (which could be forever).

Kafka's flavor of streaming stands opposed to remote procedure call like Thrift or HTTP, and to batch processing like in the Hadoop ecosystem. Unlike RPC, components communicate asynchronously: hours or days may pass between when a message is sent and when the recipient wakes up and acts on it. There could be many recipients at different points in time, or maybe no one will ever bother to consume a message. Multiple producers could produce to the same topic without knowledge of the consumers. Kafka does not know whether you are subscribed, or whether a message has been consumed. A message is simply committed to the log, where any interested party can read it.

Unlike batch processing, you're interested in single messages, not just giant collections of messages. (Though it's not uncommon to archive Kafka messages into Parquet files on HDFS and query them as Hive tables).

Case 1: Kafka does not preserve any particular temporal relationship between producer and consumer. It's a poor fit for streaming video because Kafka is allowed to slow down, speed up, move in fits and starts, etc. For streaming media, we want to trade away overall throughput in exchange for low and, more importantly, stable latency (otherwise known as low jitter). Kafka also takes great pains to never lose a message. With streaming video, we typically use UDP and are content to drop a frame here and there to keep the video running. The SLA on a Kafka-backed process is typically seconds to minutes when healthy, hours to days when healthy. The SLA on streaming media is in tens of milliseconds.

Netflix could use Kafka to move frames around in an internal system that transcodes terabytes of video per hour and saves it to disk, but not to ship them to your screen.

Case 2: Absolutely. We use Kafka this way at my employer.

Case 3: You can use Kafka for this kind of thing, and we do, but you are paying some unnecessary overhead to preserve ordering. Since you don't care about order, you could probably squeeze some more performance out of another system. If your company already maintains a Kafka cluster, though, probably best to reuse it rather than take on the maintenance burden of another messaging system.

  • 1
    Thanks @closeparen (+1) - I get most of what your saying, with one big exception. In your paragraph beginning with the sentence "Kafka's flavor of streaming stands opposed...", I'm inclined to think I could replace most instances of the word "Kafka" with "RabbitMQ", and the sentence would hold true. For RabbitMQ: producers could send a message and a consumer would pull it down and process it hours/days afterwards. Consumers can attach to a queue anytime they like, and so for RabbitMQ, there can be many different recipients at different points in time. – smeeb Jun 7 '17 at 12:20
  • 1
    Think of Kafka like a database engine with a peculiar log-oriented structure. Producers append, consumers read. Reading does not affect Kafka's state in any way. A consumer can maintain an incrementing cursor to create semantics identical to RabbitMQ pub/sub, and this is a common use case, but it's not the only use case. – closeparen Jun 7 '17 at 16:52
  • 1
    Think of RabbitMQ like a distributed version of an in-memory queue data structure. Once you pop something off a queue, it's not on the queue anymore. Sure, you might have a topology where it's been replicated to other queues for the benefit of other consumers, but you wouldn't generally be able to say "give me the message I handled 500 messages ago" or "start Queue B as a copy of Queue A from where Queue A was yesterday." – closeparen Jun 7 '17 at 16:56
  • 2
    A Kafka based system is forgiving. If you don't like how your program behaved, you can push a code change and then rewind its input. You could stop a RabbitMQ consumer without affecting producers, but you wouldn't be able to revisit the past. – closeparen Jun 7 '17 at 17:27
  • 1
    Ahhh :lightbulb: thanks (+1 for all 3)! So this is definitely a compelling case for Kafka: the ability to revisit the past. I assume there has to be some upper limit or truncation going on right? Otherwise Kafka's memory would always just be climbing up. Even if data spills over to disk, the files where topic data is stored would fill up the disk very quickly, yes? – smeeb Jun 7 '17 at 18:04
5

Kafka/Kinesis is modelled as a stream. A stream has different properties than messages.

  • Streams have context to them. They have order. You can apply window functions on streams. Although each item in a stream is meaningful, it may be more meaningful with the context around it
  • Because streams have order, you can use that to make certain statements about the semantics of processing. E.g. Apache Trident supposedly has exactly-once semantics when consuming from a Kafka stream.
  • You can apply functions to streams. You can transform a stream without actually consuming it. You can lazily consume a stream. You can skip parts of a stream.
  • You can inherently replay streams in Kafka, but you can't (without additional software) replay message queues. This is useful when you don't even know what you want to do with the data yet. It's also useful for training AI.

Generally, use Kafka for offline stream processing, use message queues for real-time client-server messages.

Example use cases from pivotal:

Kafka: Website Activity Tracking, Metrics, Log Aggregation, Stream Processing, Event Sourcing and Commit logs

RabbitMQ: general purpose messaging..., often used to allow web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures while the user waits for the result. Use when you need to use existing protocols like AMQP 0-9-1, STOMP, MQTT, AMQP 1.0

It may sometimes be useful to use both! For example in Use Case #2, if this was a stream of data from a pace-maker say, I would have pace-maker transmit heartbeat data to a RabbitMQ message queue (using a cool protocol like MQTT) where it is immediately processed to see if the source's heart is still beating. This could power a dashboard and an emergency response system. The message queue would also deposit the time series data into Kafka so that we could analyse the heartbeat data over time. For example we might implement an algorithm to detect heart disease by noticing trends in the heartbeat stream.

  • 1
    Thanks @Samuel (+1) - this is a wonderful answer and helps put things into context a little better. I actually have a few followup questions for you (if you don't mind), but they're all hinged/contingent on one initial clarification that I need: when you say "You can apply functions to streams. You can transform a stream without actually consuming it...", are those functions/transforms executed on Kafka, or do they need to be consumed first before the streams are processed via functions/transforms? – smeeb Jun 7 '17 at 12:31
  • 1
    Meaning, you have KafkaProducer, Kafka and KafkaConsumer. Let's say KafkaProducer lives inside a Java app, and that KafkaConsumer is running on some Ruby app/backend. KafkaProducer sends Message1 to Kafka that needs to be transformed via Function1. Where does Function1's code live? On Kafka (proper) or inside of KafkaConsumer (the Ruby app)? – smeeb Jun 7 '17 at 12:34
  • 2
    You can't execute functions or do any processing in Kafka itself. Apache Spark Streaming and Apache Storm are two distributed stream processing frameworks that can consume from Kafka. They run outside of Kafka and connect to it as if it were a database. The frameworks expose useful functions like splitting, aggregating, windowing, etc. You could implement basic functions in your Ruby consumer, but I would highly recommend one of the frameworks. spark.apache.org/streaming storm.apache.org/releases/2.0.0-SNAPSHOT/Trident-tutorial.html – Samuel Jun 7 '17 at 19:55
  • 1
    OK, thanks and +1 again - that would have been friggin awesome though if Kafka could do processing on the streams itself! So to play devil's advocate, couldn't you just have a RabbitMQ consumer pull down messages off a queue, aggregate them based on timestamp (or really any other criteria/attributes), and perform the same window and transform functions to the data that Spark Streaming or Storm provide? – smeeb Jun 7 '17 at 20:42
  • 1
    Yes I think you could do that with RabbitMQ because RabbitMQ has guarantees about message order. You may not be able to do it with every message queue. And it would be complex to build. E.g. what if your RabbitMQ consumer that is aggregating crashes? With Kafka, you can keep track of where in the stream you've processed up to, so you can start up your consumer at the point you left off – Samuel Jun 7 '17 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.