Microservice should be an independent software unit - Up to which level?

Question

I have read several times that a Microservice should be an independent software unit. But what does that mean exactly and is it really achievable for each business case? Does that mean that I can run a microservice stand alone without any other microservice and use the full functionality of that microservice? Or do I get the idea wrong of independent microservices?

A ProductCatalog microservice for example:

Yes this microservice can run independently. I can start it and perform different CRUD operations for my products without the need to run another microservice. Further I can also use this service to define different product categories and so on. Of course the microservice has its own database. This microservice has no dependency to other services and stands for its own.

Same goes for a Customer microservices which stores the corresponding data of the customers of the system. Further it is also responsible for the user authentication.

But what about a Checkout microservice?

This microservice performs a checkout to purchase one or more products for one user with a given paymethod. To perform a checkout the service requires a request containing the products the customer and maybe the prefered paymethod. When running this service independently it would be possible to perform a checkout for products which may not exists in the ProductCatalog as there is no dependency to the ProductCatalog microservice and therefore no verification is possible. This would or can lead to a data integrity problem. If there would be a verification against the ProductCatalog microservice the Checkout microservice would not run independently.

So my question is if this is the case when saying that microservices should be independent units of software. Or do I miss something here?

Personally, I think that the microservices should run independently. This makes testing and development much easier. However, I am concerned about data integrity and the fact that this may result in redundant data. For example the transaction history in the Check-out Service: The products there may have different or fewer data than the products in the ProductCatalog microservice Service. Since not all data is required for the checkout itself. To make this clear: Transaction history means that Customer X has bought [A,B,C] Product with Paymethod YZ.

PS: I am aware that the validation can take place in an API Gateway / Aggregator or similar. The API Gateway / Aggregator is a facade for bundling the microservice backends. The clients communicate with the API Gateway and not directly with the microservice backends.

Answer 1

I think this answer mixes up two types of dependency between services : development/deployment dependency and runtime dependency.

Development or deployment dependency would mean that each service can be designed, developed, tested, deployed and operated independently. For this, microservices should be as independent as possible. Otherwise, the advantages of scale and rapid iteration dissapear.

Runtime dependency asks what would happen, if service A depends on service B, when service B goes down. This is less clear and is a design trade-off you need to ask your business to make. In first scenario, service A has copies of everything it needs from service B. This makes sure that if service B goes down, service A can still continue working. But it causes data consistency issues, which needs to be automatically or manually detected and fixed. Other scenario has service A always check or update data from service B. This means that if service B goes down, so does service A. But it makes keeping data consistent easier. Either of those extremes are viable design. And which design is more viable depends on how your requirements around service availability and data consistency are set up.

Answer 2

When running this service independently it would be possible to perform a checkout for products which may not exists in the ProductCatalog as there is no dependency to the ProductCatalog microservice and therefore no verification is possible. This would or can lead to a data integrity problem. If there would be a verification against the ProductCatalog microservice the Checkout microservice would not run independently.

This is quite correct; and the answer to if they can truly run independently is "it depends", because transactional consistency across multiple services is very hard to do.

Suppose you are building a system where it is absolutely 100% imperative that someone cannot checkout a product if it is no longer available in the Product Catalog at the time of check-out (let's say for example this is a system for ordering critical medical supplies and you have to be absolutely sure that you don't let someone think they are getting something and then not ship it).

In that case, yes, you definitely do need to have an interdependency with the Product Catalog service and most likely, a strict separation between the two services is not the best approach. In that case you'll design your services so that the act of browsing, adding and then ckecking out products, is all contained in one larger service to ensure you can have data consistency; and you will have strict serialisation of transactions (i.e. when I checkout, I lock the stock record, I decrease the available stock, and literally nobody else can simultaneously do this until my transaction is done, thus giving me a good guarantee of data consistency)

However, let's say you are an online general goods retailer (Amazon, etc) where the scale of operations means that the potential downsides of letting someone add a product to their cart, come back 24 hours later, and checkout that cart without any reference to the existing product catalog or stock levels, are pretty small.

Then it makes sense to go with strict separation, since it means that people can load up their cart, take some time to think about it and then come back and pay you money to checkout the goods even if your main product catalog browsing service is completely offline for whatever reason. i.e. your product catalog service will indeed send messages indicating depletion of stock, etc, and these will eventually be consumed by the checkout service so that you can - as a convenience - let people know if their item is out of stock at checkout, but without any strict guarantee that you will actually know this at the time of checkout.

In that case, you are willing to bet that the few who are disappointed to get an email saying "oops, we can't ship your product, it was out of stock and we can't get any more, here is your money back" is far smaller than the many who will in practice get their products shipped.

In many ways, this mirrors reality for many products ordered online; even if your system is 100% consistent with itself, if you have a just-in-time ordering system with your supplier and I order 10 of product X from you, then you are relying on your supplier to be able to send you 10 of product X. But you have no guarantee at any point in time whilst you are dealing with me, if you can do this.

Answer 3

In my opinion, "microservice architecture" has a very specialized role ... and that it is usually deployed in the wrong situations for the wrong reasons.

A "product microservice" might be the cat's meow if you have thousands of mobile-apps out there hammering the front end of your system and you're trying to control the load on your product databases. You can construct a queue which "throttles" the incoming request-stream so that they do not become overloaded. The price you pay is both time and potential inconsistency.

Answer 4

I’ve tackled this very issue with the use of events. Every time something happens in “the system”, it fires an event to a central store. Each service that cares about that event will listen for it and do something with it. In your case Service A would fire a CustomerCreated event. Service B would fire a ProductCreated event and Service C would listen for both of them and store the data it needs to work. It then has the data it needs locally to validate your checkout. You need to break out of the thinking of a monolith for microservices to really show benefits.

Answer 5

If your architecture works against you, it's not a valid architecture.

As you have noticed, your Microservices need to behave more and more like an RDBMS to allow joining of data, and transactions across collections. CRUD Microservices are next to useless. Databases can already protect data collections, join, and coordinate transactions.

In your situation, you seem to be trapped, unable to directly query your data collections from an orchestration service:

Backwards infrastructure

Microservices kind of refers to two things. One is the architecture itself; but the other is Cloud Functions/Lambda. When people speak of "scaling" they are usually referring to the infrastructure. Infrastructure should not be used to justify architecture concepts. This is how we got into the broader n-Tier architecture mess in the first place: PHP is in the middle, therefore we have a Logic layer in between the Application and the Database. That's wrong.

Microprocesses

(I am a contributor to the following draft standard)

The Microservice Architecture is deeply flawed. Microprocess Architecture starts with a clean slate, and actually makes sense. see https://colossal.gitbook.io/microprocess/differences/compared-to-microservices.

Adapt

I'm not saying that you need to rearchitect what you have, but rather, learn from this conceptual standard. I'm not sure how you are managing your data collections; but you should start by having them in one instance (ie. Postgres or Mongo instance). This will work with the same CRUD services you already have. Then you can build an orchestration service that directly connects to the database for complex processes.

Full Migration to Microprocesses

The standard isn't complete yet, and the open source software hasn't been published. So it's best you wait [2021-01-02]. For now, the draft standard should help you understand how to architect for your own needs to more clarity.