0

Our platform is growing in the number of events we're raising, and these different events are often related to one another:

ProductCreated ProductUpdated ProductTakenOffSale ProductPutOnSale

Guaranteeing order of event consumption on a single topic isn't too difficult, but when you're faced with multiple topics like the above, you encounter some awkward edge cases when different types of event are consumed in the unintended order. For example the ordering of consumption around Products put on/off sale is very important.

Are there technologies which guarantee ordering across multiple topics for events relating to a certain type of entity? (i.e a Product in this case)

Alternatively is there some practise or style of architecture which prevents this challenge from exploding in complexity over time?

8
  • What is it about the manner of raising the events, which stops them being consumed in order? Technologies and techniques are not scarce - the problem is mostly that designers only tackle them as an afterthought. Truly distributed systems are axiomatically incompatible with the time-ordering of events, and systems capable of occasional periods of autonomy (save, say, for the occasional clock sync) are rocket science. The standard approach is just not to run a distributed system in the first place.
    – Steve
    Commented Apr 11, 2023 at 12:19
  • Certainly these concepts are easier from a monolithic application angle, but then we're faced with the difficulty of scaling an engineering department all trying to deploy the same app
    – FBryant87
    Commented Apr 11, 2023 at 12:34
  • Since when do "engineering departments" deal with matters like putting products on and off sale?
    – Steve
    Commented Apr 11, 2023 at 12:50
  • I think you may have misunderstood the context
    – FBryant87
    Commented Apr 11, 2023 at 12:53
  • 2
    I think there is misunderstanding of topics and their purpose. Topic does not mean single message type. It means stream of events related to single entity/aggregate. So you would have a topic for all events related to Product. With Product's Id serving as key for ordering. Topics are the tool you use to keep order of events you want to keep ordered.
    – Euphoric
    Commented Apr 12, 2023 at 7:26

1 Answer 1

4

An obvious solution seem to be to use a single topic for events that needs to be ordered. You could use a single ProductStatus topic, with the type of status inside the message.

Another possible solution would be to keep a version number for each product that is only increasing. So if you get a message with a lower version number than it currently has, it can be disregarded. Ofc, you would need to keep track of the current version number of all the products for this to be useful, and a central place that does the incrementing. But it will depend on specifically what you want to do. For some uses cases you might need to keep all messages to reconstruct the sequence of messages.

5
  • I'd use a timestamp in place of version. Also, you have to watch out for cases where you receive a "product deleted" before you get the "product created" which means you have to store those deletions. Lastly, your "product updated" has to be very specific on what parts of the product were updated. If those come through out of order and you don't keep track, you could accidentally undo a previous update.
    – Daniel T.
    Commented Apr 13, 2023 at 15:15
  • 1
    @DanielT. Re timestamps, I probably disagree, I think timestamps increase the risk of having out of order events, since clocks might not be synchronized, and they have limited precision. Re the rest, it depends on what you need to do. The last message is sufficient for some purposes, but you are right that some use cases require the full sequence of messages. It really depend on the actual use case.
    – JonasH
    Commented Apr 14, 2023 at 6:28
  • But how could you possibly synchronize the version numbers across multiple input systems? At least the timestamps have a chance of being in order...
    – Daniel T.
    Commented Apr 14, 2023 at 11:07
  • @DanielT. As mentioned in the answer, ordering does require some central authority of some kind, be it a service, database, or something else. But that is fundamentally unavoidable if you want to provide strong ordering guarantees. If you want completely independent inputs you need some form of conflict resolution. The only time I would trust distributed timestamps is if I could ensure all clocks are synchronized to within some margin of error, and then you need some central authority for time synchronization anyway.
    – JonasH
    Commented Apr 14, 2023 at 11:30
  • My experience is with offline mobile apps that batch up network requests for later sending. Since the source of the events is offline, there is no way to have a central authority doling out version numbers. The only solution I've thought of is timestamps, is there maybe some other solution you can think of?
    – Daniel T.
    Commented Apr 14, 2023 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.