3

I'm trying to build a distributed system and have been researching how best to manage dependencies between services. The common example in tutorials is an ordering system - let's say I have a Catalog, Order and Account service.

The Order service needs to accept an order containing a number of items, cost up the total value, do a credit check against the account, and complete the order assuming all is okay.

This single action requires data from the other two services in a synchronous fashion (e.g. we need to look up the cost of each catalog item since we can't trust the user to have not tampered with it, and we need to validate that the user has sufficient credit), and I've experimented with ways of dealing with this:

  1. Maintain a persistent cache of the required data in the Order service which is kept synchronised using messaging (e.g. events in RabbitMQ, Kafka such as itemCreated or accountUpdated.). The Order service contains a list of accounts with only the information it requires (e.g. the current credit), and a list of items (again, with a price and quantity in stock) to allow it to work autonomously. I've seen this referred to as a 'data-pump' architecture.
  2. Synchronous calls (e.g. REST/RPC) from the order service to the Catalog and Account services for it to verify the information.
  3. Another service (I think sometimes referred to as an aggregation service) sits above these three services, and aggregates the data from the Catalog and Account services into a format that the Order service can accept without doing the query itself.

Each method has issues and I'm having difficulty deciding if any are correct:

  1. A lot of data will end up being cached. In future, an order may be dependent on a new Discount service - do I cache all of these as well? My actual domain model will depend on over 8 other microservices already and it is in its infancy. Would this new Discount service also need to cache catalog items?
  2. This seems to go against some chief use-cases for independent and autonomous microservices - I've simply taken a monolithic application and swapped fast in-process calls for slower http calls. If I change one service, I likely to need to update another. If either the Catalog or Account services go offline, the Orders service cannot accept orders.
  3. This technique simply moves many of the problems with 2. to another layer.

I like the first approach due to its simplicity and resilience - if either other service goes offline then I can still make an Order. However, I am not sure I can justify the amount of data 'duplication' (I realise it's not strictly duplication) - is caching all of that information for validation (and potentially many times over for different services) acceptable practice? I can foresee the catalog being cached in many different services for example.

Thanks in advance!

2
  • You can also: 4. Cache the data in each API. That way, you will only have one point of data duplication, your responses can be quick and the best thing: you will have control over the cache. In the first option, if a catalog item has been removed, you will continue to process orders with that catalog item. But if the service catalog has control over the cache, it can delete the cache for that item when it is removed. Apr 11, 2020 at 3:23
  • Thanks Gabriel - if I understand correctly, your suggestion would entail my entire persistent storage being cached? My choice of words perhaps wasn't great - wherever I said 'cache' I was referring to persistent storage of duplicated data. I'm focusing less on performance optimisation and more on design patterns for accessing data owned by another service.
    – Laurence
    Apr 11, 2020 at 20:49

2 Answers 2

2

I'm having difficulty deciding if any are correct

Regarding option 1, I don't know what trick you're going to use to keep the caches synchronized, especially if you scale out the Order service.  But you already know this isn't microservices, really.  What if the reason a service goes offline is so you can update the products, prices & inventory — but still accept Orders based on information you know is inaccurate.  You will need to model some kind of eventual consistency and perhaps prove it is correct, I guess.

Regarding option 2, if you query a service for information (how many of some item), that information is only valid as of when it was sent — it is potentially out of date immediately, unless you can place the services in a distributed transaction (yuk!).

So, here's another way of thinking of it.

Thus, your microservices should not necessarily be merely one querying another for information later acted upon, but rather requesting commitments.  Actions like reserve some set of inventory, associated with a reservation (and maybe associated with a time out period).

Order:

  • add Order to Account (with status pending)
  • use Catalog to reserve inventory, fetch prices
  • use Account to charge total price to payment method
  • use Catalog to claim reservation (or release if payment fails)
  • use Account to change order status
    ...
1
  • 1
    Thanks Erik - I agree with your points and you've given me a different way of thinking about dependencies as actions rather than just queries for data. Whilst a better approach than 2., I don't think this will prevent me from designing what I've seen referred to as a 'distributed monolith' with a tangled web of dependencies between services. My understanding is that one of the main (if not the main) goal of microservices is independence - it seems that even in this basic use-case that is an issue.
    – Laurence
    Mar 17, 2020 at 8:59
1

To my way of thinking, this "caching" can very quickly cease to be "microservices" at all ... and maybe that's actually a good thing. Why are you using microservice architecture in this case? Why are you issuing an asynchronous SOAP-call when you could just make a subroutine-call within a single application? Microservice architecture is not a silver bullet.

If you are determined to have three services, for Catalog, Order, and Account, then I think that they should indeed be asynchronous. If they maintain caches, let each one of them maintain their own caches so that each can deliver answers faster ... but do not maintain a "client-side" cache in hope of avoiding the need to ask them, because inevitably your answer would one day become "stale" and you'd have no way to know it.

1
  • Thank you for your answer. To answer why we're considering microservices is because we've developing an extensive cloud-hosted API across a series of teams and functional areas that each have different requirements related to data storage, technology and scaling. It sounds like you are advocating for one of 2. and 3., since I think Orders caching Account and Catalog information would count as 'client-side' in your description? I would argue that a REST call is not a guarantee of avoiding stale information either but I see your point!
    – Laurence
    Mar 16, 2020 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.