1

I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach something from the past.

I split the communication into three types:

  • app to server - where I want to use simple messaging and rather do not care about anything in past as data should be proceed on-line - pushing data to the topic like events
  • server to all apps - configuration broadcast where the data is pushed to topic configurations and the app can consume from it and reach the past messages in case of any troubles
  • and the server to particular app - here I want to use topic with partitioning like configurations_per_app_id; I want to push data to the topic with proper partitioning key

My questions are - if those three approaches are correct? Especially the third one, as I didn't find any way how to deal with such use case - I found only that it is possible to use partitioning key, but what in case if there is for example 10 000 apps consuming from given topic using partition key. Is it how it should be done, or we can use some other approach base on Kafka? Is there any pattern for such needs?

  • What does "App" exactly mean? Is this a real case your system has 10 000 different type of apps? or just possible 10 000 instances of the same app? – ivenxu Oct 3 '17 at 23:16
  • By the App I mean - the application which runs on Android device and connects directly to the Kafka. I can have 10 000 devices with the same App, which has to have access to data exposed on Kafka's topics. – h__ Oct 4 '17 at 6:09
  • It's not a good idea to make any config per device simply it's not manageable. In your case, do you want your device compete each other to process the message in the queue or everyone should get a copy of the message? – ivenxu Oct 4 '17 at 9:29
  • In respect to point 2 - everyone should get copy of the message (even if now it is online). To point 3 - I want to be able to configure single device with particular configuration (even if it is offline now) – h__ Oct 4 '17 at 9:33
1

The third is not valid; all consumers on a topic get all messages. Partitions are used to spread load across multiple consumer instances (same group) and to maintain message order for specific keys.

For point-to-point messaging you need a separate topic for each app.

With JMS you can use message selectors to select messages you are interested in (as long as the publisher provides something to select on).

With RabbitMQ you can use a topic exchange and each consumer (group) binds a queue with a routing key that will select messages he has interest in.

If you don't want to use a kafka topic for each consumer, you will probably need a hybrid approach to satisfy all your use cases.

Consumers can consume from multiple topics.

  • For 3, using a separate topic or using filtering in code should suffice. – JimmyJames Nov 2 '17 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.