1

I would like to enrich some streaming data, with one service per enrichment, possibly each service written in a different language and managed/deployed separately.

Each service would consume some Kafka topic, enrich the message, and publish it in some Kafka topic.

In the end, I would have a topic of fully enriched messages.

Ideally, some of the steps would be done concurrently, but this is not a requirement.

What are frameworks, protocols, or patterns matching this kind of use cases ? To me it looks like regular async micro services, but with high risk of coupling between the services.

Improve this question
6
  • Where does the data come from, going through a pipeline and where does it ends up? – rebeling Nov 15 '16 at 19:38
  • Did you check out Kafka Streams? It is part of Kafka 0.10+ and a good fit to build micro services that read from and write to Kafka brokers, as you describe it in you question. Check out docs.confluent.io/current/streams/index.html and kafka.apache.org/documentation#streams and github.com/confluentinc/examples/tree/3.1.x/kafka-streams – Matthias J. Sax Nov 15 '16 at 21:21
  • @rebeling data comes from the message queue and goes to the message queue :) – Flow Nov 16 '16 at 8:42
  • @MatthiasJ.Sax: Things like Kafka Stream, Samza, etc would not be meet the "each service written in a different language and managed/deployed separately" requirement :) I'm actually looking for good practices or feedbacks on this kind of setup – Flow Nov 16 '16 at 8:45
  • 1
    For different languages, you are right -- even if Kafka Streams might be available in other languages at some point (but that might take a while...). But Kafka Streams applications can be managed/deployed completely decoupled from each other... Fur multi-language stream processing, there is Concord (concord.io), but it's not open source as far as I know. – Matthias J. Sax Nov 16 '16 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.