We are introducing events to our system (I would hesitate to call it 'event sourcing', but we have started down this road). To do this we are still maintaining the same HTTP public 'CRUD' API's, but on each mutation request (POST/PATCH/PUT) the server(s) will now create an event and post it to an 'event store' service. We are wondering whether a service receiving a request should update its local data first then send the event, or send the event and, having subscribed to the event store service, only update its' local data when it receives the event back?
Here is what currently happens:
A POST request comes in to
API 1 and it updates its' local data then creates an event.
API 2 is listening to these events as it needs to update its data accordingly. It receives the event and it updates its data. Great! It works quite nicely (We are also enjoying having this log of all 'mutations' in one place - great for debugging).
If we make
API 1 subscribe to the event, rather than directly update its data, I can think of the following advantages:
- If for some reason the event never makes it to the event queue service, then the data between the two API's is still in sync (API 1 did not prematurely update its local data)
- The client request can be potentially quicker, since it is not waiting on a DB update before responding
- We could put the part of API 1 that subscribes to events and updates local data in its own service, separating read and write (i.e. moving towards CQRS).
A few problems:
- API 1 cannot send a response to the client saying that the POST request is all good, since it won't know until it receives the event and updates its data. The best we can do is say that it was accepted.
- With that in mind, we now need a way of informing the client of failures - the implication seems to be we need some kind of notification service with SSE or websockets in order to push info to the client.
- The overall latency to write data goes up (its doing a full round trip)
- Often, the write API needs to access the database anyway to verify a user is allowed to do whatever they are asking (e.g. do they own the resource they want to mutate?). This can negate the advantage I said of requests being quicker; it also poses another question - should this access/permissions check be done in the initial request or when the subscriber receives the event?
Thanks for any ideas!
(I should also note that while my examples only have 2 services in play, in reality we have a number of simple CRUD API's, each managing their own data and sitting behind an API gateway. Until introducing events, we were simply calling API's directly from one another, which has become difficult to manage/debug/maintain)
In picture form, here is what we currently have, with API's updating their data then posting an event: