How to update a NoSQL database and publish event atomically in an event-driven microservices architecture?

Question

In an event-driven microservices architecture, services typically need to update their domain state and publish an integration event to a service bus at the same time (Either both operations complete, or none).

Simple example: The ordering microservice receives a REST API call to create an order. The microservice needs to:

1) Create the order entity and persist it in the database

2) Publish an OrderCreated integration event through an event bus

When using a relational database this is typically achieved using the outbox pattern: a single entry is saved in the database that indicates that event X needs to be published. This entry is saved as part of the same transaction that contains the domain-state changes. A background process then polls these entries and publishes the events. This means that the event will eventually be published, making the system eventually consistent.

However, NoSQL databases do not favor the idea of updating multiple documents in a single transaction, and many of them do not support it without ugly workarounds. Below is a list of potential solutions (Some more ugly than others):

1. Outbox pattern variation:

Outbox pattern, but instead of having a separate collection of documents for the pending events, they will be saved as part of the domain entity. Each domain entity will encapsulate a collection of events that remain to be published and a background process will poll such entities and publish the events.

Cons:

1. If the background process publishes the event but fails to remove it from the domain entity, it will re-publish it. This shouldn't really be a problem if updates are idempotent or if the event handler is able to identify duplicate events.
2. Domain entities are corrupted with integration events.

2. Event sourcing:

Event sourcing makes this problem go away but is very complex to implemented and a big overhead for small microservices.

Cons:

1. Complex, might need complete re-design of the way services work with data.

3. Listening to own events:

The service will only publish an event that is also subscribed to (It will not update its state as part of the same operation). When the service bus sends the event back for handling, the service will update its domain entity.

Cons:

1. Other microservices may handle the event before the origin microservice. This may cause problems if they assume that the event already happened when in fact it hasn't.

Are there any other solutions to this problem? Which is the best one?

Answer 1

You can't. You are writing to two different databases (NoSQL + Service Bus) - you cannot guarantee consistency between the two without introducing distributed transactions which are bound to cause headaches, slowdowns and ultimately failures.

Event sourcing is the way to go. Event sourcing is not as much of an overhead as it is believed to be, in fact, it is easy to implement (No need for frameworks, easy to build something production-ready), reduces overall complexity and increases performance for large scale applications.

Some comments about the other two approaches:

Approach #1: This seems to be a downgraded variation of ES. It is temporarily persisting the events and using the latest snapshot as the source of truth. I think this is a clear downgrade from event sourcing because:

1. History is lost.
2. Having background threads that are writing to the aggregate (To delete the events after publishing) may increase the rate of optimistic concurrency exceptions.

Approach #3: This is essentially an attempt to use the service bus as an event store - it is not meant to be an event store, don't do it. I've seen some systems that use something like Kafka as an event store, but this is because Kafka leverages events as its core principle and provides some event-sourcing features out of the box.

I recommend going through Greg Young's talks & papers about event sourcing, he's known to be one of the event sourcing gurus.

Answer 2

In an event driven structure your transaction completes when you mark the event completed; preventing its being re-executed. ie deleting it from the queue or whatever retry system you have in place.

So regardless of your db tech the process is

Pull message
do calculations
write state
post next message(s) in train
delete message

If you fail before write state you are all good, the message goes back on the queue for reprocessing.

If you write the state, but cant send the next message; well you can still put it back on the queue as long as your processor is careful about checking its state or is idempotent.

If you send the next message and then crash, that's where you have the problem. You will send the next message twice. The various queuing software have ways of deduping, but if you can expect a long delay between the first and repeat message you might want to build extra protection at the app layer, having a message Id that get passed onto subsequent messages so you can detect duplicates, timestamps so you can detect out of order processing, or use a deadletter queue and require manual processing for failed messages.

It should be noted that you have the same issues with your sql transaction 'outbox' method. If you pull a message from the 'outbox', post it but then crash before marking it done, your processor will pick it up again when it restarts

Answer 3

For adding Orestis's opinion, IMHO, how about not saving all the events in an array in a document, but just having a single event and updating it because CDC can capture all the update event using oplog even though it will be overwritten immediately by another event. If it is possible, we don't have to manage additional processes to delete published events periodically.