If I have two different producers that could produce the same message for a Kafka broker, how can I ensure that only one of the two message occurrences gets processed?

Is the only way to have an input topic, then a consumer that dedupes and saves to an output topic?

Or could the relatively new "exactly-once" functionality’s be utilized somehow for this problem?

  • 1
    Hello Syed. As it stands, your question appears to be asking for assistance in explaining, writing or debugging code. As such it is off-topic here and so is being down-voted. Please consider rewording your question to make it more on-topic. – David Arno Sep 6 '18 at 10:53
  • 1
    @DavidArno I think the question is not about programming, but about the design of an event processing stream to ensure some guarantees. So IMHO fully in scope here. Could you clerify what needs rewording ? Because the kafka terminology of topic, producer, consumer, and message seem all to be well defined. – Christophe Sep 8 '18 at 17:34

Kafka used previously only an at-least-once message handling. From the Kafka's documentation:

When publishing a message we have a notion of the message being "committed" to the log. Once a published message is committed it will not be lost as long as one broker that replicates the partition to which this message was written remains "alive".

In that scenario, nothing prevented the same producer to write the same message multiple times, for example in case of some network errors or latencies.

The new exactly-once guarantee now makes sure that if a producer writes the same message multiple times, it will be committed only once to the log. This works using a sequence number to deduplicate redundant message batches.

However, this sequence number is unique only taken together with a PID, a unique number that can be mapped from a transactional id that uniquely identifies a producer in its configuration. The detailed flows are explained in plain details in this article.

This implies that Kafka will only deduplicate identitical messages from the same producer. It will not deduplicate messages sent by two different producers. So yes, to deduplicate across several producers, you'll need a consumer that finds the same data in different messages. And this consumer will act as producer either to an additional broker, or to an additional topic on the same broker.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.