How to deal with delayed updates?

Question

I have a microservice-based system that has a product service and a store service. I'm using eventstore to propagate events between microservices, so for example when a product is created, the store service receives a notification and can then add the new product to a store.

Now I need to implement multiple stores and a user can specify whether the product is added to all stores or no stores. The product service shouldn't know about stores so adding a flag AddToAllStores to the product service would break Single Responsibility principle. But the store service is updated based on eventstore events so won't immediately know about the new product, i.e. an api in the store service to add the product to all stores will fail because the product doesn't exist yet.

The only mechanism that makes sense to me is for the store service, on receiving the request to add an unknown product to all stores is for the store service to query the product service. Is this the accepted approach for dealing with delayed updates?

Answer 1

The only mechanism that makes sense to me is for the store service, on receiving the request to add an unknown product to all stores is for the store service to query the product service. Is this the accepted approach for dealing with delayed updates?

Not necessarily.

From the point of view of doing the work, what you have inside the Store is a state machine, with effectively three states: I have the first message, I have the second message, I have both messages.

If the first message has arrived, and the second message hasn't arrived yet, then maybe we just do something else until the rest of the messages arrive. We don't have to block waiting on a remote system that may not be available - just put this item in the backlog, and work on something else until you have the rest of the data that you need.

Now, while you are waiting, you might send a message to another system, to see if the information you need is available. On the other hand, if there's enough work here already, maybe you first do the work you have, then check to see if the message was late in arriving. "It depends", what's your SLA? how much backlog is there? etc.

Doing a bunch of bookkeeping of the messages you have received is common.

Answer 2

Interesting dilemma.

The user has just created the product, and now expects to be able to add that product to their stores.

Solutions:

1. Allow the product to be added regardless.
2. The store service not having the product in its cache consults the source of truth.
3. Reject the request till such time as the system synchronises.

Off the bat option 3, is what you already have.

Option 2 is what you are currently thinking. In terms of viability it depends.

• You are adding an extra dependency, and volume which the event store was presumable meant to avoid.
• On the other hand only new products would cause the store to query the products service. Deleted and amended products would be updated via the events system so in terms of volume it might be sustainable.
• It will necessarily introduce some extra lag in when adding new products to a store, will this be a usability/user experience concern?

Option 1 uses the client as a pseudo event store by forwarding the product details via the client to the store when a product is created.

• You can achieve some reasonability here by signing those details.
  • ensure that any internal confidential data isn't sent via the client (even if encrypted).
• The event store will complete/verify the product information
• The Store has no direct dependency on the product service
• There is an increase in external bandwidth dependent on the number of new products.

I would lean toward option 1 instead of option 2 in the majority of cases. The exceptions would be when the data needed shouldn't be passed through a hostile environment.

As for the viability of Option 2 you will need metrics, and probably some extra measures around service reliability.

---

Why not simply pass a list of stores that the product should be added too?

When the user flags AddToAllStores the UI they are using should translate that to a list of all of their stores.

This translates the global knowledge problem of what does AddToAllStores mean, to the local knowledge problem of just these, stores, to, the, letter.

You will of course still need to do the translation but the right place for translation is properly in the input system be that a batch data-loading process or a human UI.