How do we maintain consistent-read promise to clients + handling ID collision when using a fallback queue?

Question

In my company, we are using Event Sourcing pattern to implement a storage for all changes to the price of a booking. Across the company, different services might try to append events to a booking identified by a booking code.

We use DynamoDB to store the event and it does support consistent read. The thing is in the case when a booking is initially made and the very 1st event is created for a booking code, if we fail to save into DynamoDB for whatever reasons, we put the event into a fallback queue and simply return a success to the client to acknowledge that we already received the event. Client can then move on with their business logic flow and in turn, show a success message to end users. The goal is to not block booking creation at all costs.

--

Problem 1: For a very short period of time, when the event is still in the fallback queue, if clients try to fetch the event using the booking code, they will get back an error although we told them that the write on the 1st event was a success earlier. In a way, we're breaking the consistent read promise here.

--

Problem 2: For the time being, only 1 department (e.g. hotel) is using this system. We're getting other departments (e.g. rental car, flight) onboard to have a single source of truth for all price changes. One problem we foresee is that there might be booking code collision as there's no guarantee that booking code is unique across department.

One approach we're trying is to first validate if the incoming booking code is unique. If it's not, we reject and ask the client to provide another booking code. If it is, we move forward with the saving into DB part which is where the fallback mechanism might kick in for new booking if failure happens.

There 2 objectives we're aiming for:

1. Make this validation mechanism works hand-in-hand with the fallback mechanism. I.e. validation should still works when we have a colliding booking code in the fallback queue.
2. Keep the Event Sourcing System oblivious of the source of events, i.e. no prepend/append client ID to the booking code as we want different departments to be able to query each other's events using only booking code.

Note: We cannot generate booking code for departments nor force them to use a company-wide unique UUID at the moment.

--

One solution I'm thinking about at the moment looks like this:

Technically, the idea is to turn my queue into a queryable topic using Kafka Compacted Topic and Kafka Stream. After sending the event to Kafka, I'll get back a stream to maintain a read-only in-memory state store of events pending in the fallback topic.

When an event arrives with version as 1 (i.e. new booking), I'll first check if the booking code already exists in the state store and return an error immediately. Otherwise, I'll do a write on DynamoDB with a where clause that makes sure there's no other events with the same booking code in the DB. If this call goes through, life is good.

If the write on DynamoDB fails because of duplicated booking code, I'll return an error to the client to request for a new one. If the write fails for other reasons, I'll try to send the event to Kafka. If this operation also fails, I return an error to the client for client-side error handling (e.g. retry).

The problem I have with this design is that if I have a cluster of Event Sourcing System, I might get hundreds of new booking events at peak hours and duplicated booking code might arrive at different nodes in the cluster at the same time.

Let's say Event 1 arrives at Node A, validation passes as there's no duplicated code. Execution proceeds to insert to DynamoDB and fails so fallback mechanism kicks in to send to Kafka. When Kafka returns an acknowledgement, I return success to client 1. Supposed while the whole Kafka process is happening and Node B is not yet aware of an incoming event from Kafka Stream, Event 2 arrives at Node B and validation also passes so it proceeds to insert to DynamoDB and the call goes through.

At this point, I have a problematic event in the fallback topic that I cannot drop as I already return a success to client 1.

--

I'm trying to find a way where we can improve the current design and keep the consistent-read promise while handling collision properly and remaining out of the way of the main booking flow (i.e. not blocking the booking on failure).

I'd be very grateful if someone could give me an idea on how to improve my solution. If you have better suggestion, I'd love to hear it too.

Answer 1

I would have a process which immediately processes the (hopefully short) fallback queue, writes the required info to a non-distributed database and then posts the message to the main fallback queue for further processing.

You can then add a fallback query to this central database in the case that the code isn't found.

Hopefully at any given time you only have a small number of these anomalous first messages, so although this adds a central point of failure to your system, it is fast enough not to affect overall performance.

If the query is being made by the same client as the initial request, ie i make a booking and then immediately load it on the next page, then I would remove the second request altogether. If you have the information locally just use it.

Regarding your duplicate key problem, rather than attempt to search your distributed db and various queues for duplicates, which would be an inherent bottleneck, use a unique generation strategy such as giving the client reserved lists of ids, composite ids which include the client id, or one of the "short guid" algorithms