0

I'm trying to use Kafka for my event handling, but I'm struggling a bit to understand what pattern I should use.

My use case is this:

  • One consumer waits for one type of event.
  • When that event is processed, the same consumer need to produce several different types of events to different topics using a subset of that incoming events properties.
  • Then once all of those events are consumed (the order doesn't matter) by their respective consumer I need to know it and send the aggregated response to another topic.

I've looked at building a Saga state machine, event aggregator, fan in/fan out and scatter/gather as possible solutions, but I'm not sure what would be the best approach. Kafka Streams?

Any suggestions would be highly appreciated.

Improve this question
4
  • please don't cross-post: stackoverflow.com/questions/75687977/… "Cross-posting is frowned upon as it leads to fragmented answers splattered all over the network..."
    – gnat
    Commented Mar 9, 2023 at 17:44
  • Regardless of patterns, Kafka Streams does the first two bullets perfectly fine using filter function. What exactly do you mean by "send the aggregated response back." Back to what?
    – OneCricketeer
    Commented Mar 10, 2023 at 14:38
  • @OneCricketeer I updated the question to say "to another topic" instead of back. My bad. Yeah, Streams seem to do the job, unfortunately I discovered that there isn't a reliable lib for Kafka Streams for .NET, so I'm out of luck. Maybe ksqldb instead, any opinon about that?
    – Chris Klingsater
    Commented Mar 13, 2023 at 8:06
  • I don't have .NET experience (I believe I found a project on Github for porting Kafka Streams to C#)... ksqlDB can work, yes, but not as flexible
    – OneCricketeer
    Commented Mar 13, 2023 at 20:05

1 Answer 1

Reset to default
1

Yeah you need a routing service that listens for completion of the various sub processes and then posts a "completed" message.

This means that process is holding the state in memory while it waits, which seems problematic but should be fine. Kafka's non-messagequeuey hang on to all the messages approach should work well here; as you can loop back from the start of all the "done" and"complete" messages to rebuild the current state in the event of a crash.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.