2

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B creates an entity of its own based on that event, and C makes a synchronous call to D to perform an action. Finally D receives both A and B's entity and performs the action requested by C which requires both A and B's entities.

Note: A,B, and D are essentially CRUD services which also have REST APIs, while C is business logic and has no state. A, B, and D are intended to be product-agnostic services in a cloud environment.

This flow works fine when dealing with updating existing entities as service D stores a partial copy of the data it receives from A and B, but it creates a sort of race condition when A creates a new entity. Note that even if A and B have created their entities before C performs the call, there's no guarantee that D has read both events.

What are common ways of dealing with this? I've come up with several, but none of them seem particularly great.

  1. Retry Pattern
    • Which entity performs the retry? Since this action needs to eventually be performed, I assume having it in service D is a bad idea as it then can't distinguish from bad input through the REST API, or requires duplicated functionality to differentiate between an event and a REST call.
    • Even if that was acceptable, it pushes business logic into a CRUD service.
  2. Webhook between D and B
    • Again, puts the logic into a CRUD service (kinda).
  3. Thread sleep on C
    • These anywhere in a running service generally seems like a horrible idea...
    • Doesn't help with the condition where B goes down while the rest stay up.

Edit in response to answers:

  1. Have C wait for the two Events before it invokes D
    • This unfortunately doesn't guarantee that D has read those same events, even though it is more likely.
    • C can't send the full contents of the event from A with the request, just the ID of the object without duplicating the REST interface methods on D; other services to come won't have anything except the ID when they request D's service(s). In all other cases, all the necessary information will reside in D to perform its service, the awkward case seems to be only for new entities.
  2. Have D wait for both create events from A and B and then do the logic without C having to tell it to.
    • This seems to break the single-responsibility principle.
    • A, B, and D are intended to be 'product-agnostic' services, see edit to the note above. The implication is that this would push logic into a service that is used by multiple products, even though the logic is specific to only one product.
1

Think of your situation as data and processes that depend on it. What results is this:

  • Service A creates entity a
  • Service B creates entity b
  • Service D performs a process d which depends on a and b
  • The decision on when d should happen is made by service C

D has to do its duty when C tells it to, but C does not provide all dependencies for that process.


Thats the problem how i see it. Now on to the solution: If you want to keep D as simple as possible, C has to provide all necessary information to D. There is no way around that.

If it is acceptable to put some logic into D (depending on the language and frameworks in use, that logic might happen to be 2 LoC), you can wait for all data in D:

  1. Entity a is created with ID 11
  2. Entity b is created with ID 22 and referes to a#11
  3. C listens for the create-event of b and then tells D the following: "As soon as you have a#11 and b#22, do d with them.
  4. D waits for both events and then does its thing.

If you are using tools like Promises and ReactiveX, this becomes a really simple thing:

// within C
class EntityEvents
    Observable<EntityCreatedEvent<T>> onCreated(Class<T> entityClass)

entityEvents.onCreated(EntityB.class).subscribe(b -> d.doYourThing(b.aId, b.id));

// within D
class EntityRepository
    Promise<EntityA> getExistingOrOnCreated(long aId)
    Promise<EntityB> getExistingOrOnCreated(long bId)

// within D - when the call "doYourThing" from C is received
Promise.all(
   entityRepository.getExistingOrOnCreated(aId),
   entityRepository.getExistingOrOnCreated(bId)
).then((entityA, entityB) -> {
   // do whatever it is you need to do
})

This gets a lot harder when you have to persists the waits for the entities. In that case, a scheduled job may be the better choice.

  • Thanks; it looks like I have a couple more details to add to the question that might (or might not) change your answer. See the 'edit'. – Asmodean May 14 '17 at 13:22
  • @Asmodean i updated my answer – marstato May 14 '17 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.