I'm evaluating the Kafka. It provides some nice features which I want to use, especially durability be default and the access to past messages for new consumers, whenever the consumer wants to reach something from the past.
I split the communication into three types:
- app to server - where I want to use simple messaging and rather do not care about anything in past as data should be proceed on-line - pushing data to the topic like events
- server to all apps - configuration broadcast where the data is pushed to topic configurations and the app can consume from it and reach the past messages in case of any troubles
- and the server to particular app - here I want to use topic with partitioning like configurations_per_app_id; I want to push data to the topic with proper partitioning key
My questions are - if those three approaches are correct? Especially the third one, as I didn't find any way how to deal with such use case - I found only that it is possible to use partitioning key, but what in case if there is for example 10 000 apps consuming from given topic using partition key. Is it how it should be done, or we can use some other approach base on Kafka? Is there any pattern for such needs?