1

I have kafka consumer that get data from kafka and do some process if some circumstances meet and send it for another consumer to do it's job.

How can i build something like this?

One way I think about is consumer-1 get data from a topic-1 process it and send result to topic-2 for consumer-2 to consume. Is there any better way to do this like using kafka Streams??

1

You’ve pretty much got it. If you really have a need to have two separate services processing the data this way, then that’s exactly how you approach it.

producer -> topicA -> service1 -> topicB -> service2

Since service1 is both reading and writing to Kafka, it is both a consumer and a producer. That’s really all the Kafka Streams library is. It’s an abstraction for cases where you’re both a consumer and producer.


Just a quick note about separating the processing into two separate services this way:

There are definitely pros and cons to this architecture, so consider it carefully. If you decide to have multiple services handle this pipeline of processing, you need to deploy and manage more services. There’s a definitive cost to that. That cost may be outweighed by the ability to reprocess parts of your pipeline or by the ability to deploy updates to different parts of the pipeline independently.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.