I've created about a dozen services for a intranet. Now I've gotten to the point that these services are more coupled to each other than I'm comfortable with, and I've been having problems where once service can cause a few other service to degrade.
I've been working on a EDA design so that I can make these services decoupled from each other but as I'm the only programmer here I need some feedback.
The design aims to solve these scenarios:
Scenario 1: Client requests data from Service A, to fulfill that request Service A needs data from Service B but Service B has failed and is unresponsive
Scenario 2: Client updates an entity in Service B, the reference to this entity needs to be updated in Service A and C
I've split the design into two parts, Operations and Events
An operation is sent by a service or client to change an entity in a service. This always results in an event being fired. Each operation contains a user id, operation id and a few timestamps.
An event is a reaction to a operation, it notifies anyone who's listening that a change has occurred. Each event contains the id of the operation that started it, the id of the user that caused it and some timestamps. Each event also has a hash of the complete entity so that any listeners can compare against their versions.
I'm using RabbitMQ as a message bus and each service has persistent durable queues to store any pending events or operations.
In order to decouple service from each other, each service caches any entities it depends on that belong to another services.
When a service requires a entity from another service it fetches it and stores it in a table in its database. It then listens for any changes made to any entities in its cache and updates them when an event related to it occurs.
When storing or fetching an entity from the cache fails, it will re-fetch it from the appropriate service.
For auditing, I've got a service that listens to events from all services and stores them. It can then recreate any entity at any given point in time.
I'd love comments from anyone more experienced than I am and whether there are any glaring issues with the design.