How can I handle multi-producer to particular single-consumer in Kafka?

Question

I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach something from the past.

I split the communication into three types:

• app to server - where I want to use simple messaging and rather do not care about anything in past as data should be proceed on-line - pushing data to the topic like events
• server to all apps - configuration broadcast where the data is pushed to topic configurations and the app can consume from it and reach the past messages in case of any troubles
• and the server to particular app - here I want to use topic with partitioning like configurations_per_app_id; I want to push data to the topic with proper partitioning key

My questions are - if those three approaches are correct? Especially the third one, as I didn't find any way how to deal with such use case - I found only that it is possible to use partitioning key, but what in case if there is for example 10 000 apps consuming from given topic using partition key. Is it how it should be done, or we can use some other approach base on Kafka? Is there any pattern for such needs?

Answer 1

The third is not valid; all consumers on a topic get all messages. Partitions are used to spread load across multiple consumer instances (same group) and to maintain message order for specific keys.

For point-to-point messaging you need a separate topic for each app.

With JMS you can use message selectors to select messages you are interested in (as long as the publisher provides something to select on).

With RabbitMQ you can use a topic exchange and each consumer (group) binds a queue with a routing key that will select messages he has interest in.

If you don't want to use a kafka topic for each consumer, you will probably need a hybrid approach to satisfy all your use cases.

Consumers can consume from multiple topics.