Event sourcing: Update local data before posting an event?

Question

We are introducing events to our system (I would hesitate to call it 'event sourcing', but we have started down this road). To do this we are still maintaining the same HTTP public 'CRUD' API's, but on each mutation request (POST/PATCH/PUT) the server(s) will now create an event and post it to an 'event store' service. We are wondering whether a service receiving a request should update its local data first then send the event, or send the event and, having subscribed to the event store service, only update its' local data when it receives the event back?

Here is what currently happens: A POST request comes in to API 1 and it updates its' local data then creates an event.

API 2 is listening to these events as it needs to update its data accordingly. It receives the event and it updates its data. Great! It works quite nicely (We are also enjoying having this log of all 'mutations' in one place - great for debugging).

If we make API 1 subscribe to the event, rather than directly update its data, I can think of the following advantages:

• If for some reason the event never makes it to the event queue service, then the data between the two API's is still in sync (API 1 did not prematurely update its local data)
• The client request can be potentially quicker, since it is not waiting on a DB update before responding
• We could put the part of API 1 that subscribes to events and updates local data in its own service, separating read and write (i.e. moving towards CQRS).

A few problems:

• API 1 cannot send a response to the client saying that the POST request is all good, since it won't know until it receives the event and updates its data. The best we can do is say that it was accepted.
• With that in mind, we now need a way of informing the client of failures - the implication seems to be we need some kind of notification service with SSE or websockets in order to push info to the client.
• The overall latency to write data goes up (its doing a full round trip)
• Often, the write API needs to access the database anyway to verify a user is allowed to do whatever they are asking (e.g. do they own the resource they want to mutate?). This can negate the advantage I said of requests being quicker; it also poses another question - should this access/permissions check be done in the initial request or when the subscriber receives the event?

Thanks for any ideas!

(I should also note that while my examples only have 2 services in play, in reality we have a number of simple CRUD API's, each managing their own data and sitting behind an API gateway. Until introducing events, we were simply calling API's directly from one another, which has become difficult to manage/debug/maintain)

In picture form, here is what we currently have, with API's updating their data then posting an event:

And here is what we are considering, with API's only updating data in response to events:

Answer 1

We are wondering whether a service receiving a request should update its local data first then send the event, or send the event and, having subscribed to the event store service, only update its' local data when it receives the event back?

The usual answer is to pay attention to your failure modes - what happens if we power down the service, erasing all of its transient memory, at the least convenient moment?

We want to make sure that information is durably stored in the book of record first (so that it is available after the restart), and then you let all of the side effects happen. The argument here being that if you have captured the information, the side effects can be reconstructed.

---

A common pattern, especially in relational data stores, is to store events in the "same" database as the book of record - you update both the book of record and the list of events in the same transaction. That's fine, because there's no room for a fault in between, so you don't run into the consistency problems you might have if the events are written "somewhere else".

(When you get into event sourcing, the list of events is the book of record, so you are good to go).

So if you were to arrange your second diagram so that API 2 was reading events out of the API 1 database, or if the event store were pulling events out of the API 1 database, then you would be "fine" because there's no risk that of sharing information that hasn't been correctly stored for later use by the API 1 service.