Does AWS Kinesis guarantee a specific type of ordering?

Question

From AWS documentation:

A Kinesis data stream is an ordered sequence of data records. Each record in the stream has a sequence number that is assigned by Kinesis Data Streams

I don't see any connection between those 2 sentences. Also, the documentation doesn't explicitly specify what type of ordering it is.

After a small investigation, I found out that if we are using AWS producer library (for example, KPL), the order which the lib successfully sent to Kinesis records is maintained when consuming from Kinesis. The problem is, this order of records doesn't guarantee to be equal to the order of records I sent to any library.

Application -(1)> KPL -(2)> Kinesis

The order is determined by the sequence of records which successfully sent from KPL to Kinesis. (In case of network failures, a record will be sent from KPL again, after other records will be sent).

---

A definition of the word randomly is here:

Of or relating to an event in which all outcomes are equally likely...

My conclusion is that an application sends records to kinesis in a random order which means, there is no order at all. In case an application must guarantee any type of order, it must create its own library and not use theirs.

Am I missing something here?

Answer 1

The term that I would use is "arbitrary" rather than "random". But your observation is correct: the sequence numbers merely indicate the sequence in which records were added to the partition. They are intended to allow consumers to continue reading where they left off (using an AFTER_SEQUENCE_NUMBER shard iterator), not to reflect the producer's sense of order. The problem gets more difficult when you introduce multiple shards and distribute your writes using partition keys.

This is an issue with any distributed system where there are multiple paths from source to destination: you can't guarantee ordering through the system (well, not without a lot of effort).

So the solution is to record ordering before putting records into the system. Assuming that your application is able to maintain order right up to the point that records are written to the stream (and this is rarely true!) then you can add a sequence number to each record (this has to be part of the payload written to Kinesis).

Then the reader must re-apply order based on those sequence numbers. This is not a completely intractable problem: records that were ordered prior to being written to the stream will usually be close to each other when read from the stream (thus arbitrary order, not random). So you must maintain records in memory until you've accumulated an unbroken sequence, then pass them on to whatever is processing them.