Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job.
How can i build something like this?
One way I think about is consumer-1 get data from a topic-1 process it and send result to topic-2 for consumer-2 to consume. Is there any better way to do this like using kafka Streams??
You’ve pretty much got it. If you really have a need to have two separate services processing the data this way, then that’s exactly how you approach it.
producer -> topicA -> service1 -> topicB -> service2
Since service1 is both reading and writing to Kafka, it is both a consumer and a producer. That’s really all the Kafka Streams library is. It’s an abstraction for cases where you’re both a consumer and producer.
Just a quick note about separating the processing into two separate services this way:
There are definitely pros and cons to this architecture, so consider it carefully. If you decide to have multiple services handle this pipeline of processing, you need to deploy and manage more services. There’s a definitive cost to that. That cost may be outweighed by the ability to reprocess parts of your pipeline or by the ability to deploy updates to different parts of the pipeline independently.
