Synchronous communcation vs caching in microservices

Question

I'm trying to build a distributed system and have been researching how best to manage dependencies between services. The common example in tutorials is an ordering system - let's say I have a Catalog, Order and Account service.

The Order service needs to accept an order containing a number of items, cost up the total value, do a credit check against the account, and complete the order assuming all is okay.

This single action requires data from the other two services in a synchronous fashion (e.g. we need to look up the cost of each catalog item since we can't trust the user to have not tampered with it, and we need to validate that the user has sufficient credit), and I've experimented with ways of dealing with this:

1. Maintain a persistent cache of the required data in the Order service which is kept synchronised using messaging (e.g. events in RabbitMQ, Kafka such as itemCreated or accountUpdated.). The Order service contains a list of accounts with only the information it requires (e.g. the current credit), and a list of items (again, with a price and quantity in stock) to allow it to work autonomously. I've seen this referred to as a 'data-pump' architecture.
2. Synchronous calls (e.g. REST/RPC) from the order service to the Catalog and Account services for it to verify the information.
3. Another service (I think sometimes referred to as an aggregation service) sits above these three services, and aggregates the data from the Catalog and Account services into a format that the Order service can accept without doing the query itself.

Each method has issues and I'm having difficulty deciding if any are correct:

1. A lot of data will end up being cached. In future, an order may be dependent on a new Discount service - do I cache all of these as well? My actual domain model will depend on over 8 other microservices already and it is in its infancy. Would this new Discount service also need to cache catalog items?
2. This seems to go against some chief use-cases for independent and autonomous microservices - I've simply taken a monolithic application and swapped fast in-process calls for slower http calls. If I change one service, I likely to need to update another. If either the Catalog or Account services go offline, the Orders service cannot accept orders.
3. This technique simply moves many of the problems with 2. to another layer.

I like the first approach due to its simplicity and resilience - if either other service goes offline then I can still make an Order. However, I am not sure I can justify the amount of data 'duplication' (I realise it's not strictly duplication) - is caching all of that information for validation (and potentially many times over for different services) acceptable practice? I can foresee the catalog being cached in many different services for example.

Thanks in advance!

Answer 1

I'm having difficulty deciding if any are correct

Regarding option 1, I don't know what trick you're going to use to keep the caches synchronized, especially if you scale out the Order service.  But you already know this isn't microservices, really.  What if the reason a service goes offline is so you can update the products, prices & inventory — but still accept Orders based on information you know is inaccurate.  You will need to model some kind of eventual consistency and perhaps prove it is correct, I guess.

Regarding option 2, if you query a service for information (how many of some item), that information is only valid as of when it was sent — it is potentially out of date immediately, unless you can place the services in a distributed transaction (yuk!).

So, here's another way of thinking of it.

Thus, your microservices should not necessarily be merely one querying another for information later acted upon, but rather requesting commitments.  Actions like reserve some set of inventory, associated with a reservation (and maybe associated with a time out period).

Order:

• add Order to Account (with status pending)
• use Catalog to reserve inventory, fetch prices
• use Account to charge total price to payment method
• use Catalog to claim reservation (or release if payment fails)
• use Account to change order status
  ...

Answer 2

To my way of thinking, this "caching" can very quickly cease to be "microservices" at all ... and maybe that's actually a good thing. Why are you using microservice architecture in this case? Why are you issuing an asynchronous SOAP-call when you could just make a subroutine-call within a single application? Microservice architecture is not a silver bullet.

If you are determined to have three services, for Catalog, Order, and Account, then I think that they should indeed be asynchronous. If they maintain caches, let each one of them maintain their own caches so that each can deliver answers faster ... but do not maintain a "client-side" cache in hope of avoiding the need to ask them, because inevitably your answer would one day become "stale" and you'd have no way to know it.