In my company, we are using Event Sourcing pattern to implement a storage for all changes to the price of a booking. Across the company, different services might try to append events to a booking identified by a booking code.
We use DynamoDB to store the event and it does support consistent read. The thing is in the case when a booking is initially made and the very 1st event is created for a booking code, if we fail to save into DynamoDB for whatever reasons, we put the event into a fallback queue and simply return a success to the client to acknowledge that we already received the event. Client can then move on with their business logic flow and in turn, show a success message to end users. The goal is to not block booking creation at all costs.
Problem 1: For a very short period of time, when the event is still in the fallback queue, if clients try to fetch the event using the booking code, they will get back an error although we told them that the write on the 1st event was a success earlier. In a way, we're breaking the consistent read promise here.
Problem 2: For the time being, only 1 department (e.g. hotel) is using this system. We're getting other departments (e.g. rental car, flight) onboard to have a single source of truth for all price changes. One problem we foresee is that there might be booking code collision as there's no guarantee that booking code is unique across department.
One approach we're trying is to first validate if the incoming booking code is unique. If it's not, we reject and ask the client to provide another booking code. If it is, we move forward with the saving into DB part which is where the fallback mechanism might kick in for new booking if failure happens.
There 2 objectives we're aiming for:
- Make this validation mechanism works hand-in-hand with the fallback mechanism. I.e. validation should still works when we have a colliding booking code in the fallback queue.
- Keep the
Event Sourcing Systemoblivious of the source of events, i.e. no prepend/append client ID to the booking code as we want different departments to be able to query each other's events using only booking code.
Note: We cannot generate booking code for departments nor force them to use a company-wide unique UUID at the moment.
One solution I'm thinking about at the moment looks like this:
Technically, the idea is to turn my queue into a queryable topic using
Kafka Compacted Topic and
Kafka Stream. After sending the event to
Kafka, I'll get back a stream to maintain a read-only in-memory
state store of events pending in the fallback topic.
When an event arrives with version as 1 (i.e. new booking), I'll first check if the booking code already exists in the
state store and return an error immediately. Otherwise, I'll do a write on
DynamoDB with a where clause that makes sure there's no other events with the same booking code in the DB. If this call goes through, life is good.
If the write on
DynamoDB fails because of duplicated booking code, I'll return an error to the client to request for a new one. If the write fails for other reasons, I'll try to send the event to
Kafka. If this operation also fails, I return an error to the client for client-side error handling (e.g. retry).
The problem I have with this design is that if I have a cluster of
Event Sourcing System, I might get hundreds of new booking events at peak hours and duplicated booking code might arrive at different nodes in the cluster at the same time.
Let's say Event 1 arrives at Node A, validation passes as there's no duplicated code. Execution proceeds to insert to
DynamoDB and fails so fallback mechanism kicks in to send to
Kafka returns an acknowledgement, I return success to client 1. Supposed while the whole
Kafka process is happening and Node B is not yet aware of an incoming event from
Kafka Stream, Event 2 arrives at Node B and validation also passes so it proceeds to insert to
DynamoDB and the call goes through.
At this point, I have a problematic event in the fallback topic that I cannot drop as I already return a success to client 1.
I'm trying to find a way where we can improve the current design and keep the consistent-read promise while handling collision properly and remaining out of the way of the main booking flow (i.e. not blocking the booking on failure).
I'd be very grateful if someone could give me an idea on how to improve my solution. If you have better suggestion, I'd love to hear it too.