I have an architectural question. I have an application which is subscribed to a log compacted Kafka topic. I have to process each event and store into a persistent datastore. I am planning to run the app in 4 instances with the same group-id so that the partitions are distributed with all the nodes.

During the lifetime of the application, there will be a need for

  • Resetting consumer to start from 0th offset
  • Pausing the consumer for a while
  • Resuming a paused consumer

The best way to achieve this was to implement an API which can be called from a script. The API will have an access to the KafkaConsumer object and will call the pause operation.

Now, the issue I see with this approach is there isn't a way I know which can ensure all of the 4 instances. How can I make sure that one API (or N API where N = number of instances) call will do this operation?

Any help on this would be appreciated.


If your problem is to basically control all of the instances of Kafka processing nodes, well, you already have Kafka. Why not introduce a "command" topic which all nodes subscribe to? You can send one command from your "api" that all the nodes will receive (as long as they have different group ids).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.