3

In my company, we are using Event Sourcing pattern to implement a storage for all changes to the price of a booking. Across the company, different services might try to append events to a booking identified by a booking code.

We use DynamoDB to store the event and it does support consistent read. The thing is in the case when a booking is initially made and the very 1st event is created for a booking code, if we fail to save into DynamoDB for whatever reasons, we put the event into a fallback queue and simply return a success to the client to acknowledge that we already received the event. Client can then move on with their business logic flow and in turn, show a success message to end users. The goal is to not block booking creation at all costs.

--

Problem 1: For a very short period of time, when the event is still in the fallback queue, if clients try to fetch the event using the booking code, they will get back an error although we told them that the write on the 1st event was a success earlier. In a way, we're breaking the consistent read promise here.

--

Problem 2: For the time being, only 1 department (e.g. hotel) is using this system. We're getting other departments (e.g. rental car, flight) onboard to have a single source of truth for all price changes. One problem we foresee is that there might be booking code collision as there's no guarantee that booking code is unique across department.

One approach we're trying is to first validate if the incoming booking code is unique. If it's not, we reject and ask the client to provide another booking code. If it is, we move forward with the saving into DB part which is where the fallback mechanism might kick in for new booking if failure happens.

There 2 objectives we're aiming for:

  1. Make this validation mechanism works hand-in-hand with the fallback mechanism. I.e. validation should still works when we have a colliding booking code in the fallback queue.
  2. Keep the Event Sourcing System oblivious of the source of events, i.e. no prepend/append client ID to the booking code as we want different departments to be able to query each other's events using only booking code.

Note: We cannot generate booking code for departments nor force them to use a company-wide unique UUID at the moment.

--

One solution I'm thinking about at the moment looks like this:

enter image description here

Technically, the idea is to turn my queue into a queryable topic using Kafka Compacted Topic and Kafka Stream. After sending the event to Kafka, I'll get back a stream to maintain a read-only in-memory state store of events pending in the fallback topic.

When an event arrives with version as 1 (i.e. new booking), I'll first check if the booking code already exists in the state store and return an error immediately. Otherwise, I'll do a write on DynamoDB with a where clause that makes sure there's no other events with the same booking code in the DB. If this call goes through, life is good.

If the write on DynamoDB fails because of duplicated booking code, I'll return an error to the client to request for a new one. If the write fails for other reasons, I'll try to send the event to Kafka. If this operation also fails, I return an error to the client for client-side error handling (e.g. retry).

The problem I have with this design is that if I have a cluster of Event Sourcing System, I might get hundreds of new booking events at peak hours and duplicated booking code might arrive at different nodes in the cluster at the same time.

Let's say Event 1 arrives at Node A, validation passes as there's no duplicated code. Execution proceeds to insert to DynamoDB and fails so fallback mechanism kicks in to send to Kafka. When Kafka returns an acknowledgement, I return success to client 1. Supposed while the whole Kafka process is happening and Node B is not yet aware of an incoming event from Kafka Stream, Event 2 arrives at Node B and validation also passes so it proceeds to insert to DynamoDB and the call goes through.

At this point, I have a problematic event in the fallback topic that I cannot drop as I already return a success to client 1.

--

I'm trying to find a way where we can improve the current design and keep the consistent-read promise while handling collision properly and remaining out of the way of the main booking flow (i.e. not blocking the booking on failure).

I'd be very grateful if someone could give me an idea on how to improve my solution. If you have better suggestion, I'd love to hear it too.

3
  • 1
    how short is very short? could you just delay the response a few ms?
    – Ewan
    Oct 7 at 12:38
  • @Ewan that's a good question. Actually, I think I made the wrong statement. We do have a retry process in place to consume from the queue. However, how long some event remains stuck in the queue entirely depends on how long we have trouble with DynamoDB. If Amazon does a good job, it should be short. I took a look at 1 year statistics, we have less than 300 events ever arrived at the fallback queue. Chance of failure is low but we want to use the same system for more services internally going forward. So I'm looking for a way to improve our promise to the clients.
    – JamesBoyZ
    Oct 7 at 13:37
  • I guess I worry when you start to try and fix something that your underlying system is supposed to do. If the write is always supposed to work but doesnt, will your remedy be any more reliable? Rather than write a fall back for "can't write to db for obscure reason" you might be better writing a "flip to Azure when AWS is down" for those times when something major goes down
    – Ewan
    Oct 7 at 13:47
1

I would have a process which immediately processes the (hopefully short) fallback queue, writes the required info to a non-distributed database and then posts the message to the main fallback queue for further processing.

You can then add a fallback query to this central database in the case that the code isn't found.

Hopefully at any given time you only have a small number of these anomalous first messages, so although this adds a central point of failure to your system, it is fast enough not to affect overall performance.

If the query is being made by the same client as the initial request, ie i make a booking and then immediately load it on the next page, then I would remove the second request altogether. If you have the information locally just use it.

Regarding your duplicate key problem, rather than attempt to search your distributed db and various queues for duplicates, which would be an inherent bottleneck, use a unique generation strategy such as giving the client reserved lists of ids, composite ids which include the client id, or one of the "short guid" algorithms

7
  • Thanks for the suggestion. I realized I missed some key information in my post so I updated it with clearer description. I hope you'll find the problems interesting.
    – JamesBoyZ
    Oct 7 at 18:04
  • As for your suggestion, I think the non-distributed DB might be a critical point of failure. I.e. if for some reason, I cannot insert into this DB from the 1st queue, I won't have events available in the 2nd queue for retry. In addition, while the events are stuck on the 1st queue, I also cannot query to find out colliding booking code.
    – JamesBoyZ
    Oct 7 at 18:06
  • im not up on kafka, but it sounds like the "compacted topic" is essentially a non-distributed db. If you cant process the first queue then its no different from any other queue processing failure, the message remains on the queue, or you push it somewhere else.
    – Ewan
    Oct 7 at 18:16
  • The difference is if I can send the event into Kafka, it means Kafka is running. And the Event Sourcing System, the one who sent the event (i.e. running) is also the one who's receiving the stream back. So there's no additional point of failure. Who should be up and running is up and running. If Kafka fails, the process fails early and client is notified of the failure. If Kafka acknowledges and the event get streamed into multiple nodes as state store, the node which receives a request can do validation in-memory.
    – JamesBoyZ
    Oct 7 at 18:20
  • 1
    i think that's a bit of a false premise, surely there are error types where you compacted topic can be broken while other queues are working in a Kafka cluster. You have an extra point of failure per working part, regardless of the technology used.
    – Ewan
    Oct 7 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.