I have an architectural question. I have an application which is subscribed to a log compacted Kafka topic. I have to process each event and store into a persistent datastore. I am planning to run the app in 4 instances with the same group-id so that the partitions are distributed with all the nodes.
During the lifetime of the application, there will be a need for
- Resetting consumer to start from 0th offset
- Pausing the consumer for a while
- Resuming a paused consumer
The best way to achieve this was to implement an API which can be called from a script. The API will have an access to the KafkaConsumer object and will call the
Now, the issue I see with this approach is there isn't a way I know which can ensure all of the 4 instances. How can I make sure that one API (or N API where N = number of instances) call will do this operation?
Any help on this would be appreciated.