1

I'm interested in integrating with an external system which uses webhooks to notify clients of events. The system is very similar to Stripe - the REST endpoints have rate limits to avoid undue polling load and instead relies on clients/users receiving webhooks to keep track of state. In the rest of this post, I'll write as if I'm integrating with Stripe since it's similar and familiar.

I'm interested in what it would take to build this webhook integration as some sort of event stream. I'm imagining that I would be able to save each event I receive into my database (Postgres) or stream (Kafka) and then build out some form of projection to aggregate these events into views I care about. For example, having a projection of all products purchased by each user.

Normally, I feel like I would be able to handle this quite easily. However, one issue I run into is that Stripe (and the system I'm integrating with) doesn't:

  1. guarantee the order of delivery of webhook events
  2. guarantee high fidelity timestamping of each event - events delivered have a created_at attribute which is a measure of seconds since epoch (not milli or nanoseconds)

For example, the Stripe docs mention

Stripe doesn’t guarantee delivery of events in the order in which they’re generated. For example, creating a subscription might generate the following events:

  • customer.subscription.created
  • invoice.created
  • invoice.paid
  • charge.created (if there’s a charge)

Your endpoint shouldn’t expect delivery of these events in this order, and needs to handle delivery accordingly.

These events may all be delivered with the same value of created_at which means I can't rely on that field to order the events.

Is there a well known design pattern for handling aggregations of events which are not guaranteed to be in order?

One idea I have is to provide my projections with a list of events that occurred with, say, 3 seconds, and let the projection do various permutations of the order to figure out which is "correct". That being said, I know this idea isn't scalable (what happens if 100 events are generated with 1 second) and is quite brittle (what happens if there are issues in with the upstream service?).

2
  • 1
    If specific fields in each event do not communicate the order they occurred, then it seems like the only fallback is to reorder events based on the problem domain. Can a payment event happen before an order created event? That is an example question that popped into my head. Is there some sort of transaction Id common to these events? Commented Aug 6, 2023 at 14:41
  • I assume that in the upstream system these events will be created in the correct order. However, it's not guaranteed that the events will be delivered to my system in the "right" order. Does that answer your question? Reordering based on the problem domain is an interesting idea - what happens if a set of events that are all related are delivered "very far" apart because of, for example, network issues? Commented Aug 7, 2023 at 10:44

1 Answer 1

3

Is there a well known design pattern for handling aggregations of events which are not guaranteed to be in order?

As far as I can tell, the usual answer is "don't do that" - if you need a sequence of events that are in the same order that they appear in the event store, then make an ordered copy of the information in the event store, rather than trying to reconstruct the sequence from isolated pieces.

What will make sense in some systems is to embed in the metadata of the event either an internal clock (see lamport-1978) or some other means of tracking "happens-before" (ex: "causation identifier").

Another possibility in some systems is that you can ignore this problem altogether, because all of the events "commute", guaranteeing the same final state for any ordering of events (see kleppmann-2020)

2
  • How would I "make an ordered copy of the information in the event store" when the events sent to my event store don't have a clear mechanism to order by? Commented Aug 7, 2023 at 10:45
  • By replacing your current storage mechanism with one that has appropriate storage guarantees. Commented Aug 7, 2023 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.