Designing event-driven architecture for multiple services

Question

I've created about a dozen services for a intranet. Now I've gotten to the point that these services are more coupled to each other than I'm comfortable with, and I've been having problems where once service can cause a few other service to degrade.

I've been working on a EDA design so that I can make these services decoupled from each other but as I'm the only programmer here I need some feedback.

The design aims to solve these scenarios:

Scenario 1: Client requests data from Service A, to fulfill that request Service A needs data from Service B but Service B has failed and is unresponsive

Scenario 2: Client updates an entity in Service B, the reference to this entity needs to be updated in Service A and C

I've split the design into two parts, Operations and Events

An operation is sent by a service or client to change an entity in a service. This always results in an event being fired. Each operation contains a user id, operation id and a few timestamps.

An event is a reaction to a operation, it notifies anyone who's listening that a change has occurred. Each event contains the id of the operation that started it, the id of the user that caused it and some timestamps. Each event also has a hash of the complete entity so that any listeners can compare against their versions.

I'm using RabbitMQ as a message bus and each service has persistent durable queues to store any pending events or operations.

In order to decouple service from each other, each service caches any entities it depends on that belong to another services.

When a service requires a entity from another service it fetches it and stores it in a table in its database. It then listens for any changes made to any entities in its cache and updates them when an event related to it occurs.

When storing or fetching an entity from the cache fails, it will re-fetch it from the appropriate service.

For auditing, I've got a service that listens to events from all services and stores them. It can then recreate any entity at any given point in time.

I'd love comments from anyone more experienced than I am and whether there are any glaring issues with the design.

Answer 1

I got a couple of things I'd like to point out:

1. I'd refer to commands instead of operations. That fits the lingo around the system you're modeling better I think.
2. It seems like your using the events as a trigger to retrieve data from other services. Will you also use the events to create those entities in the first place? If that's the case, I'd strongly suggest to introduce some form of EventStore for each application to store the events in which you're publishing. You can then reuse those events to recreate your entities if necessary. Most people refer to this as Event Sourcing, although I prefer to use that term for the write side (the 'operation handling' in your example) of the application. Nonetheless, I'd create your entities on those events as well. If you'd take that approach, you could even let the other services listen to all those events and create the entity they'd need themselves, which will omit the need to query the entities.
3. You point out you'll add another service for auditing. If you follow my suggestion from point 2, your event store will automatically also be your audit log. This will of course only hold if you model your events correctly, but nonetheless I think making your events first class citizens and store them immediately after publishing omits the need of a dedicated audit-log service.
4. Without knowing what programming language you are in, I'd still suggest taking a look at some CQRS/EDA/DDD frameworks out there, like the Axon Framework for example. It's Java based, but even if you're not on Java, the concepts the framework tries to help you with resemble what you're trying to do.

All over, I think that remodeling your application set to something like you're suggestion is a good thing. Making your communication message based and your services location transparent ensure that you lose the coupling you're worried about at this moment.

Answer 2

First of all, if service A needs data from service B, then your service boundaries are wrong. It seems that the criteria you used for splitting your system is by entity. It correlates to layered services concept, which was already considered evil back in 2007.

The approach I'm advocating is for is based on the concept of business capability. You identify some areas in your domain that are really very cohesive, that comprise of some encapsulated data used only within that area, business-processes implemented there, applications, and people. Then you delve deeper in those areas and repeat that step. I find it convenient to use a value-chain analysis for that. Consider you prime business-process. How do you make money, what's the main reason of your business' existence? Take a look at this process as a sequence of steps. If some step contains some business-processes that are not connected with any other steps, than probably these steps can be a standalone service.

Your technical services map to those business-services as 1:1.

There is a series of posts on SOA that contain some examples as well, check it out.