2

In an event-driven microservices architecture, services typically need to update their domain state and publish an integration event to a service bus at the same time (Either both operations complete, or none).

Simple example: The ordering microservice receives a REST API call to create an order. The microservice needs to:

1) Create the order entity and persist it in the database

2) Publish an OrderCreated integration event through an event bus

When using a relational database this is typically achieved using the outbox pattern: a single entry is saved in the database that indicates that event X needs to be published. This entry is saved as part of the same transaction that contains the domain-state changes. A background process then polls these entries and publishes the events. This means that the event will eventually be published, making the system eventually consistent.

However, NoSQL databases do not favor the idea of updating multiple documents in a single transaction, and many of them do not support it without ugly workarounds. Below is a list of potential solutions (Some more ugly than others):

1. Outbox pattern variation:

Outbox pattern, but instead of having a separate collection of documents for the pending events, they will be saved as part of the domain entity. Each domain entity will encapsulate a collection of events that remain to be published and a background process will poll such entities and publish the events.

Cons:

  1. If the background process publishes the event but fails to remove it from the domain entity, it will re-publish it. This shouldn't really be a problem if updates are idempotent or if the event handler is able to identify duplicate events.
  2. Domain entities are corrupted with integration events.

2. Event sourcing:

Event sourcing makes this problem go away but is very complex to implemented and a big overhead for small microservices.

Cons:

  1. Complex, might need complete re-design of the way services work with data.

3. Listening to own events:

The service will only publish an event that is also subscribed to (It will not update its state as part of the same operation). When the service bus sends the event back for handling, the service will update its domain entity.

Cons:

  1. Other microservices may handle the event before the origin microservice. This may cause problems if they assume that the event already happened when in fact it hasn't.

Are there any other solutions to this problem? Which is the best one?

1

In an event driven structure your transaction completes when you mark the event completed; preventing its being re-executed. ie deleting it from the queue or whatever retry system you have in place.

So regardless of your db tech the process is

Pull message
do calculations
write state
post next message(s) in train
delete message

If you fail before write state you are all good, the message goes back on the queue for reprocessing.

If you write the state, but cant send the next message; well you can still put it back on the queue as long as your processor is careful about checking its state or is idempotent.

If you send the next message and then crash, that's where you have the problem. You will send the next message twice. The various queuing software have ways of deduping, but if you can expect a long delay between the first and repeat message you might want to build extra protection at the app layer, having a message Id that get passed onto subsequent messages so you can detect duplicates, timestamps so you can detect out of order processing, or use a deadletter queue and require manual processing for failed messages.

It should be noted that you have the same issues with your sql transaction 'outbox' method. If you pull a message from the 'outbox', post it but then crash before marking it done, your processor will pick it up again when it restarts

  • So the microservice will simply insert an entry to the database, and the bacgkround handler will be responsible for both 1) Updating the state and 2) Publishing the event? In the case of a REST API call, doesn't that mean that the caller will receive a 200 OK response even before the state is actually updated? – J. Doe Jun 24 at 12:29
  • no. if your initial input is a rest call write the state and push events to message queues. if something fails return 500 and get user to resend – Ewan Jun 24 at 12:50
  • or better. just push a message for later processing – Ewan Jun 24 at 12:51
  • Maybe I'm missing something, but if I write the state and then the event fails to be published, then the state of the system will be inconsistent because other microservices were not notified. I can return 500 to the user but there's no guarantee that the user will re-try the operation, leaving the state modified. – J. Doe Jun 24 at 13:17
  • you can leave your commit untill after sending – Ewan Jun 24 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.