Event Driven Microservice Race Condition

Question

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B creates an entity of its own based on that event, and C makes a synchronous call to D to perform an action. Finally D receives both A and B's entity and performs the action requested by C which requires both A and B's entities.

Note: A,B, and D are essentially CRUD services which also have REST APIs, while C is business logic and has no state. A, B, and D are intended to be product-agnostic services in a cloud environment.

This flow works fine when dealing with updating existing entities as service D stores a partial copy of the data it receives from A and B, but it creates a sort of race condition when A creates a new entity. Note that even if A and B have created their entities before C performs the call, there's no guarantee that D has read both events.

What are common ways of dealing with this? I've come up with several, but none of them seem particularly great.

1. Retry Pattern
   • Which entity performs the retry? Since this action needs to eventually be performed, I assume having it in service D is a bad idea as it then can't distinguish from bad input through the REST API, or requires duplicated functionality to differentiate between an event and a REST call.
   • Even if that was acceptable, it pushes business logic into a CRUD service.
2. Webhook between D and B
   • Again, puts the logic into a CRUD service (kinda).
3. Thread sleep on C
   • These anywhere in a running service generally seems like a horrible idea...
   • Doesn't help with the condition where B goes down while the rest stay up.

Edit in response to answers:

1. Have C wait for the two Events before it invokes D
   • This unfortunately doesn't guarantee that D has read those same events, even though it is more likely.
   • C can't send the full contents of the event from A with the request, just the ID of the object without duplicating the REST interface methods on D; other services to come won't have anything except the ID when they request D's service(s). In all other cases, all the necessary information will reside in D to perform its service, the awkward case seems to be only for new entities.
2. Have D wait for both create events from A and B and then do the logic without C having to tell it to.
   • This seems to break the single-responsibility principle.
   • A, B, and D are intended to be 'product-agnostic' services, see edit to the note above. The implication is that this would push logic into a service that is used by multiple products, even though the logic is specific to only one product.

Answer 1

Think of your situation as data and processes that depend on it. What results is this:

• Service A creates entity a
• Service B creates entity b
• Service D performs a process d which depends on a and b
• The decision on when d should happen is made by service C

D has to do its duty when C tells it to, but C does not provide all dependencies for that process.

---

Thats the problem how i see it. Now on to the solution: If you want to keep D as simple as possible, C has to provide all necessary information to D. There is no way around that.

If it is acceptable to put some logic into D (depending on the language and frameworks in use, that logic might happen to be 2 LoC), you can wait for all data in D:

1. Entity a is created with ID 11
2. Entity b is created with ID 22 and referes to a#11
3. C listens for the create-event of b and then tells D the following: "As soon as you have a#11 and b#22, do d with them.
4. D waits for both events and then does its thing.

If you are using tools like Promises and ReactiveX, this becomes a really simple thing:

// within C
class EntityEvents
    Observable<EntityCreatedEvent<T>> onCreated(Class<T> entityClass)

entityEvents.onCreated(EntityB.class).subscribe(b -> d.doYourThing(b.aId, b.id));

// within D
class EntityRepository
    Promise<EntityA> getExistingOrOnCreated(long aId)
    Promise<EntityB> getExistingOrOnCreated(long bId)

// within D - when the call "doYourThing" from C is received
Promise.all(
   entityRepository.getExistingOrOnCreated(aId),
   entityRepository.getExistingOrOnCreated(bId)
).then((entityA, entityB) -> {
   // do whatever it is you need to do
})

This gets a lot harder when you have to persists the waits for the entities. In that case, a scheduled job may be the better choice.

Answer 2

If D requires Entities from A and B, simply event notifies C only, C is the logic to control the rest of the flow so C calls A, then C calls B, then C calls D.