0

Let's say I want to use CosmosDB for Event Store, where have single collection for all events. Currently it is not horizontally partitioned, but it is a possibility for future.

Now, I need to implements event handlers, ideally as Azure Functions, but AppService will work well as well.

Do you have some guide, how to implement the messaging between event store and event handlers reliably?

I need to have guarantee, that

  1. Every event that is saved to the DB will be delivered to all event handlers once and exactly once (not zero, not two times)

  2. When replaying the events, they have to be processed in the same order as when they were first handled.

Of course, I would like to keep the latency between event store and event handlers as low as possible.

What is my problem?

The requirement #1 disqualifies all event store implementations, that relies on pushing events to some message bus, because that operation is not granular and some events could possibly stay undelivered.

In my current solution is each event handler keeps track of last processed event,

I would maybe pull the events from event source (the pull could be triggered by Cosmos DB triggers) and then deliver to the event handlers, but then event handlers needs to keep track of what events have been handled and I don't want to do this on my own from scratch.

The other problem is the event orders. Since there is no built in auto incremented column, how to guarantee, that I retrieve the events always in the same order?

  • Given events T1, T2a and T2b
  • When T2a and T2b have the same timestamp
  • When query at time T2 returns events T1,T2a,
  • Then the same query at later tile won't return T1, T2b, T2a...

Does ComsosDB support this out of the box?

0

Every event that is saved to the DB will be delivered to all event handlers once and exactly once

Surrender, Dorothy. Trying to achieve that is not going to be unreliable. The network is not reliable.

You may be able to get close to what you want by pairing At-Least-Once delivery (the sender keeps broadcasting the message until it receives an acknowledgement) with Idempotent event handling (the event handler does something sensible when a duplicate message arrives).

(Note: in the general case, idempotent message handling requires that you keep track of every message ever, which for large message volumes will start to lead to scaling problems. You can beat the scaling problem if you are willing to lose some messages. Trade offs.)

Event ordering is a little bit easier - each handler writes down the order in which messages arrive. That way, the messages are always processed in the same order. If you look at the presentations on the LMAX Disruptor, you'll see this idea expressed there.

Of course, the order that things are received is not necessarily the same as the order that things are sent; which order is the authoritative one?

Ordering things that happen concurrently is a pain. You can look into Lamport clocks, or modeling happens-before relationships in your messages.

You may find that using a pull, rather than a push, model for messaging simplifies a lot of your ordering problems. See Greg Young's talk on polyglot data.

  • Hmm, what about event ordering on the command stack, i.e. getting events for particular aggregate. Here is the ordering even more important. We can assume that single aggregate events are not partitioned... – Liero Apr 11 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.