What are my options if I want to make my Kafka consumer idempotent (due to Kafka's at-least-once semantics) if my project's only database is DynamoDB, which is eventually consistent?

I'm afraid setting the "consistent read" flag might slow down and overload the leader node;

I have neither control over the topics I consume nor over the Kafka cluster.

  • 1
    Perhaps an in-memory cache of message hashes with an object expiration slightly longer than the DynamoDB propagation lag? I'm not in your head, so it's difficult to know how many options you really have. – Dan Wilson Oct 30 at 13:15
  • Yeah, I understand what you mean @DanWilson. This is a good approach and in the end it has to be about understanding the trade-offs and consequences we can live with. Thanks. – Ra Yokai Oct 30 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.