3

I know this question came up by a lot of people here and around the Internet but I couldn't manage to find a really clear explanation how to do queries spanning multiple microservices.

Imagine that I have 2 services, one is managing user relations, and one is responsible for blog posts. If I just want to get the 20 latest posts, I have to query the relations database for ALL my friends/follows. That could be a thousand entries. Then I pass these to the blog service and it returns 20 blog posts. Then I scroll down a bit and (imagine that I don't cache) this operation is repeated. This is an overkill in my opinion.

Most of the answers to these questions stated that the separation of domains is not good, and if these two services are always used together then they belong to one service. That's also unacceptable for me because most of us are just writing our programs in microservice way, but all the features are somehow coherent. So in the end if I'd have many joins I'd always be building monoliths? Or should I accept the inefficiency if I go on the microservice path?

3

"most of us are just writing our programs in microservice way". Be careful with this statement, because what "microservice way" is is definitely not a single way and I would say that many times it is done wrong. The reason for this is that many developers create microservices based on entities: User, Blog, Comment, Category, Payment, etc. This sort of design creates a web of relationships where all services have dependencies on all other services and the requirements of creating queries with joins on data from multiple services appear.

So, I would say, the first thing you should do is: accept that your service boundaries are likely to be wrong. A service should be able to perform its business goals without requiring data from other services. I have a strong feeling that this is your situation. This means that you should be able to do a query in a single service and return a list of result ids. Then use this ids to query one other or multiple service to create the full information that needs to be returned to the user (for example, the user name, the user reputation and the text of the blog post will likely live in 3 different services).

Now, if you are 100% sure that your service boundaries are correct and you still find a situations where you need to do queries across multiple services, you have a couple of options:

  1. Create a search engine: This could be firing several searches in parallel to multiple services and combining the results, or doing a search in one service and then call other services to filter out the results from the previous one.
  2. Use a Search service: this service aggregates data from multiple services and indexes it in a way that allows efficient searches. This is useful if you want to do complex queries like full text search on all blogs and comments, searches based on multiple categories or tags, users, etc. These services normally sort the results based on % of match. If the searches are complex, it's best to use a third party service and not try to build your own.

Regarding your last question "Or should I accept the inefficiency if I go on the microservice path?" I would say that a properly designed microservices application shouldn't be more inefficient than a monolith, considering "efficiency" not only the speed of a single query, but a property of the whole application (performance, stability, mantainability, etc). If your microservice design is not better than your monolith design, definitely go for the monolith.

0

If you need a join on the data of two Microservices, your Microservices design is simply not correct. Data that logically works together in your system belongs in the same Microservice.

Another alternative is to duplicate the data of the user microservice into other Microservices as needed using events E.g. relations into posts (but not addresses) and addresses into billing (but not relations). Note that this way you can end up with a distributed (tm) big ball of mud (tm) pretty fast.

  • If we see each "bounded context" as an independent sum of service and data, whether the data is duplicated or not is irrelevant, so we should not be worried about ending up with a "big ball of mud". A big ball of mud is hard and expensive to undo and involves other elements, However, MS (and their data) must be possible to be deleted (or take it down) without causing issues to the "whole". Any time. When you think in that "duplicated data" as "different representation of the data" it's easier for you to design each MS independent from each other, regardless of the data. – Laiv Sep 3 '18 at 6:59
  • @Laiv that entirely depends on how you share that data and how your workflows look like. I don't say you necessarily end up with a BBOM. But I've seen it happen quite a large amount of times – marstato Sep 3 '18 at 8:03
  • 1
    Well, to my understanding, MS don't share data. But I've seen it happen quite a large amount of times probably because the same error we all do the first time we face this architecture: we split the code, instead of design bounded contexts. Many people (Including myself) only focus on splitting the code and strive to apply rules like DRY which are counterproductive in MS environments (and leads to hat BBOM) but put no efforts on understanding how the business works and how to bring those mechanics to the architecture. It takes a whole new way to focus and see the design when it comes to MS. – Laiv Sep 3 '18 at 8:08
  • @Laiv if one set of data is replicated to another microservice (e.g. user relations into the posts service) that is, technically speaking, not "sharing" data. But you have to keep it up to date; so, semantically, i'd still call this a one-directional sharing. – marstato Sep 3 '18 at 8:45
  • But you have to keep it up to date it depends on the needs. It's not a rule of thumb. If we can not provide services totally separated, isolated and agnostic each other, then MS is not the architecture we need. Many developers only see the "split code into small pieces" and forget the concept "choreography". If we only focus on splitting code (technically) and ways to make parts aware of each other (technically), we are fated to fail terribly at implementing this architecture. – Laiv Sep 3 '18 at 9:15
0

Define a view database, which is a read-only replica that is designed to support that query. The application keeps the replica up to date by subscribing to Domain events published by the service that own the data.

https://microservices.io/patterns/data/cqrs.html

-1

If you blog service accepts a list of userIds as input, ie

select posts where userid in (1,2,3,4)

then there is no particular inefficiency.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.