1

I'm trying to check my understanding of the kappa architecture. The main point I am trying to resolve is how, if at all, one unifies the processing of data under analysis and other pieces of data, such as a user changing a setting which could later affect data analysis.

For example, suppose IoT data of electrical usage in homes is being sent to an immutable log like Kafka. We wish to perform various analytics on said data to produce some views of the data for end users (how much electricity you are using, cohort comparisons, etc).

Now suppose that there are settings an administrator can dial up and down to determine thresholds for some analytics. If I want to enable true data replay I would need to capture these changes as well. So my log would end up something like this.

D=data event
S=settings event
T=time

  [D5,S3,S2,D4,D3,D2,S1,D1,D0,S0]
Tn<------------------------------T0

So events D{0,1} are processed using S0 settings. Then the user tweaks the settings and S1 is generated and added to the log (and eventually processed). D{2,3,4} are processed under the new settings changes from S1. Etc.

If I wanted to replay everything and regenerate my views, I would need to process the data in order. And since the data events depend on the settings events, it would seem to me that I could not parallelize the processing.

Is my understanding correct? My understanding of the kappa architecture guidance is that it doesn't necessarily prescribe any particular approach to processing the data - that mostly depends on your use case, yes?

I'm trying to wrap my head around how I might extract efficiencies from using this approach but still guarantee in order processing, especially since new events may depend on previous events.

At least in the scenario I outlined, I do not think there is a pattern by which I could parallelize processing and still guarantee in order processing.

Update 30-Mar-2017 So this was never answered, but the comments are good enough to discern one possible architecture. I would also suggest to anyone interested to check out this older article from Microsoft on CQRS and Event Sourcing.

https://msdn.microsoft.com/en-us/library/jj554200.aspx

Yes, it is from 2012 but the fundamentals are pretty good.

  • You can do a stream-to-stream process that turns D4,S2,D3,D2,S1,D1,S0 into (D4,S2),(D3,S1),(D2,S1),(D1,S0). At this point the data items would be independent. (You don't actually have to materialize this into a new Kafka stream if you don't want.) Workers can then just grab intervals of the second log however and process them. Would this resolve the issue for you? If not, why not? – Derek Elkins Feb 14 '17 at 7:53
  • Take D5, from my example. It depends on the outcome of S2 and then S3. This wasn't stated in my original example statement, but S3 and S2 do not necessarily cancel each other out. S2 could be a manipulation of setting #1 and S3 could be a manipulation of setting #42, So if the current state of the system is S={...,S2,S3,...} then after S2 event it would be S={...,S2',S3,...} and after S3 event it would be S={...,S2',S3',...}. I also do not think that spitting them necessarily makes them independent. D4 could be from the same device as D1. They need to be processed in order. – Mike Feb 14 '17 at 10:32
  • Okay, it wasn't clear what independence assumptions you intended. In the extreme case of every piece of data depending on the all earlier data, sequential would be the best you could do. This helps clarify though. (As for the "settings", doing a running aggregation into a "dictionary" would likely be no issue in practice, so that distinction isn't too critical.) – Derek Elkins Feb 14 '17 at 11:37
  • Yeah, apologies for not being clear in the original question. As I read your comment it occurred to me that I had left some important details out. Thank you for commenting. – Mike Feb 14 '17 at 11:41
  • If one is not concerned with replay-ability and is ok with settings changes eventually taking affect (such that you could not guarantee that S1 would necessarily affect the processing of D{2..} but only that it would eventually be applied at some future point against D{x..} x>=2) then I think you could split the settings events into separate stream and achieve some efficiency there. But the vast majority of events will be of type D. – Mike Feb 14 '17 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.