How to build an event sourced system with AWS (Specifically, how to deal with side effects)

Question

So the current setup is a typical event sourced system built on AWS: Lambdas exposed with API Gateway to get the orders (commands), the domain (aggregates) apply some validations after loading the aggregate, publish all events in the event store (DynamoDB).

Where things get interesting! How to react to these events! Some of these events might have direct implications, some are side effects!

Implementation A:

We started with having the direct implication (for example, if we have an order, after running the validation we will have an event something like OrderConfirmed and then this event will be used to trigger the rest of the flow! Like _actually_ ordering the item) implemented as lamdbas that listen directly to the DynamoDB Stream! Sounds amazing! Especially with how DynamoDB Streams will collect the events by batches and trigger the lamdba.

The issue with that implementation is that DynamoDB Stream does NOT filter events! Which means if I have flow A and flow B, depending on which event I raise I want my system to react differently, Lambda A and Lambda B will be triggered in both cases, it will be my job to have a filter in these lamdbas, which is obviously not nice to have! And thats the case of the main flow not the side effects (like notifications/emails)

Implementation B:

Let's have a dispatcher lamdbda that will filter everything for us! At first it sounded like a good idea, 1 dispatcher lambda, that will publish all events to an SNS (filtered by topics!) and have all other lambdas to subscribe based on this topic! Everything is wonderful but we just created a Single Point Of Failure

Implementation C:

We decided that maybe, main implications shouldn't be handled by the event but directly by the domain! So for example, when publishing orderConfirmed, directly we call the order service to start preparing the order! Rather than having an event handler for orderConfirmed. But the issue here is, how do we deal with side effects then? Do we go back to implementation A or B for the side effects?

In the end

I'm not experienced much with AWS ecosystem, I dealt with this issue before not in a serverless setup, the application had a context and I used a framework that had its own implementation of an event bus like, and having handlers is more straightforward and made more sense. I'm looking at AWS EventBridge since it provides something similar to that but not sure if it's the right place.

Answer 1

I believe that the usual practice is to write messages to SNS topics at the time you put interesting changes into DyanmoDB.

If I'm understanding correctly this is a hybrid of your ideas B and C. Topics, to keep the publisher and subscribers decoupled, but with the messages published to the topics by the same process that writes to your event store.

You can think of it as having your domain code publish events to two different audiences: one audience is itself in the future (these go to dynamodb), the other audience is everybody else (these go to SNS for fanout).

It's not a complete solution (you still have to worry about things like "what happens if there is a crash after we've written to dynamodb but before we published to the last SNS topic?")

Answer 2

As mentioned in the In the end part, I decided to use EventBridge, as it gives a lot of freedom towards what to trigger and how. Also you can benefit from using different event busses for different events.

The idea is that, after pushing any kind of event to my event store, I would go ahead and push it also to my event bridge.

I used the filter functionality to pick which lambda to trigger based on the event type which makes everything very straightforward and easy.

The drawbacks of this method is that, what if you were able to push the event to the event store but not to your event bridge? You will end up with an inconsistency. There is a couple of ways to resolve this but the drawback is there nonetheless