1

In dealing with distributed data across Microservices, common solutions I've read for sharing this data are:

a) using a local cache where the cache is updated from domain events coming from the service that owns that data,

b) creating a materialized view, updating via domain events or database triggers.

Say we have services for Orders and Customers and we introduce a new Coupons service that needs to initialize all Customers with a Coupon based on their Sales data. How does this new service get all the existing Sales and Customer data when it's first launched? Is it standard to seed the cache or materialized view with some data loading script first, before switching to event- or trigger-based updates after that?

1 Answer 1

1

Streams - Events and Snapshots

Most modern filesystems contain an event log. Everytime a file is altered, created, deleted, moved, etc... a new log entry is added with the next id.

A program can monitor the state of the file-system by looking at this log and noting the new entries from the last entry it read. Using its own internal data representation and those updates it can keep current with the file-system.

Sometimes though the last log entry the program read has fallen off the log. So it cannot be sure of what else has dropped off the queue. The only way to restore state is to read a snapshot of the file system from a recent point in time, guessing what happened in the filesystem. Then catch up with the latest events (hopefully the log hasn't overflowed again).

So yes it does make sense to perform a bulk-load, or load from a complete source representation. However I wouldn't have another process responsible for this, it is the programs job itself to manage its own state, it should know how to get itself going from a cold start, or a long slumber.

4
  • If the program (e.g. Coupon service) manages how to get itself going - if it's required to ensure it has all customer data it could end up calling the customer service REST API 1,000's or 10,000's of times to extract all the data it needs? Since it shouldn't have direct access to the Customer database. You're suggesting the Customer service should have an event log, and the Coupon service reads from that?
    – DLT
    Feb 4, 2021 at 3:50
  • yes and no. The event Stream is part of the API. It just that your service maintains a local cache/view of that service that allows it to answer most questions without checking. The events can be pulled by, or pushed to that service. I would invest in both so that you get the snappyness of a push, but still ensure currency through pulls. If the cache becomes too old or the question is too sensitive fall back to direct querying.
    – Kain0_0
    Feb 4, 2021 at 7:02
  • So the source API (Customer) emits an event stream that is subscribed to by the Coupon service, so events are pushed for fast consistency. But the Coupon service can also pull from the event stream - is that where you need a technology like Kafka to hold the event stream and you can replay from a given bookmark (or the start)? And the fall back to direct query could be REST, gRPC, etc?
    – DLT
    Feb 5, 2021 at 9:16
  • Depends entirely on what is in the customer service. If it is supports an event stream internally, then like a file system it does not require an external technology, simply expose it via the API. If it doesn't, and just generates events then its still a choice. Without an intermediary missed events will cause a full reload of the data from the client (to ensure currency). If that rate of failure is too high then an intermediary might make sense.
    – Kain0_0
    Feb 7, 2021 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.