3

While using Apache Kafka, is it possible to get the offset of the message that has been produced?

From the Kafka documentation page it says:

each message in the partition are assigned a unique sequential id/number called Offset

Now suppose when I am producing any message I want to retrieve the offset that has been assigned to it.

  • @Spoike thanks for adding the new tag, much appreciated. – user2720864 Sep 6 '13 at 12:48
2

You shouldn't do that from a class that acts as a producer, like in most queue systems, as its responsibility is to just fire and forget the messages. The broker will do the appropriate meta data handling with id's, offsets, whatever it needs to handle the messages in a correct manner.

You can get the offsets as a consumer of messages from a Kafka broker. If you look in the SimpleConsumer class, you'll notice it fetches MultiFetchResponse objects that contains offsets as a list. Furthermore when you iterate the Kafka messages you'll end up with a MessageAndOffset objects that contains both the message sent and it's offset.

  • 1
    Agreed, but why I am asking this because I want to keep a track of the message offset that has been produced. I then might able to select that offset to read from in the consumer end. Does this make sense? – user2720864 Sep 6 '13 at 12:06
  • @user2720864 why do you want to select just a particular message at the producer using offsets? you can select messages with using topics instead. – Spoike Sep 6 '13 at 12:19
  • Not a particular message but I want to consume starting from that offset – user2720864 Sep 6 '13 at 12:31
  • 1
    You don't need to use an offset for this. You can do this by just starting the consumer before sending messages. – Spoike Sep 6 '13 at 13:20
2

The producer response protocol provides the offset of the first message sent in the messageset. Provided the API/Implementation you're using exposes this information, you should be able to retrieve it from the response to producing your message. More here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.