Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B creates an entity of its own based on that event, and C makes a synchronous call to D to perform an action. Finally D receives both A and B's entity and performs the action requested by C which requires both A and B's entities.
Note: A,B, and D are essentially CRUD services which also have REST APIs, while C is business logic and has no state. A, B, and D are intended to be product-agnostic services in a cloud environment.
This flow works fine when dealing with updating existing entities as service D stores a partial copy of the data it receives from A and B, but it creates a sort of race condition when A creates a new entity. Note that even if A and B have created their entities before C performs the call, there's no guarantee that D has read both events.
What are common ways of dealing with this? I've come up with several, but none of them seem particularly great.
- Retry Pattern
- Which entity performs the retry? Since this action needs to eventually be performed, I assume having it in service D is a bad idea as it then can't distinguish from bad input through the REST API, or requires duplicated functionality to differentiate between an event and a REST call.
- Even if that was acceptable, it pushes business logic into a CRUD service.
- Webhook between D and B
- Again, puts the logic into a CRUD service (kinda).
- Thread sleep on C
- These anywhere in a running service generally seems like a horrible idea...
- Doesn't help with the condition where B goes down while the rest stay up.
Edit in response to answers:
- Have C wait for the two Events before it invokes D
- This unfortunately doesn't guarantee that D has read those same events, even though it is more likely.
- C can't send the full contents of the event from A with the request, just the ID of the object without duplicating the REST interface methods on D; other services to come won't have anything except the ID when they request D's service(s). In all other cases, all the necessary information will reside in D to perform its service, the awkward case seems to be only for new entities.
- Have D wait for both create events from A and B and then do the logic without C having to tell it to.
- This seems to break the single-responsibility principle.
- A, B, and D are intended to be 'product-agnostic' services, see edit to the note above. The implication is that this would push logic into a service that is used by multiple products, even though the logic is specific to only one product.