Microservice JOIN query

Question

I know this question came up by a lot of people here and around the Internet but I couldn't manage to find a really clear explanation how to do queries spanning multiple microservices.

Imagine that I have 2 services, one is managing user relations, and one is responsible for blog posts. If I just want to get the 20 latest posts, I have to query the relations database for ALL my friends/follows. That could be a thousand entries. Then I pass these to the blog service and it returns 20 blog posts. Then I scroll down a bit and (imagine that I don't cache) this operation is repeated. This is an overkill in my opinion.

Most of the answers to these questions stated that the separation of domains is not good, and if these two services are always used together then they belong to one service. That's also unacceptable for me because most of us are just writing our programs in microservice way, but all the features are somehow coherent. So in the end if I'd have many joins I'd always be building monoliths? Or should I accept the inefficiency if I go on the microservice path?

Answer 1

"most of us are just writing our programs in microservice way". Be careful with this statement, because what "microservice way" is is definitely not a single way and I would say that many times it is done wrong. The reason for this is that many developers create microservices based on entities: User, Blog, Comment, Category, Payment, etc. This sort of design creates a web of relationships where all services have dependencies on all other services and the requirements of creating queries with joins on data from multiple services appear.

So, I would say, the first thing you should do is: accept that your service boundaries are likely to be wrong. A service should be able to perform its business goals without requiring data from other services. I have a strong feeling that this is your situation. This means that you should be able to do a query in a single service and return a list of result ids. Then use this ids to query one other or multiple service to create the full information that needs to be returned to the user (for example, the user name, the user reputation and the text of the blog post will likely live in 3 different services).

Now, if you are 100% sure that your service boundaries are correct and you still find a situations where you need to do queries across multiple services, you have a couple of options:

1. Create a search engine: This could be firing several searches in parallel to multiple services and combining the results, or doing a search in one service and then call other services to filter out the results from the previous one.
2. Use a Search service: this service aggregates data from multiple services and indexes it in a way that allows efficient searches. This is useful if you want to do complex queries like full text search on all blogs and comments, searches based on multiple categories or tags, users, etc. These services normally sort the results based on % of match. If the searches are complex, it's best to use a third party service and not try to build your own.

Regarding your last question "Or should I accept the inefficiency if I go on the microservice path?" I would say that a properly designed microservices application shouldn't be more inefficient than a monolith, considering "efficiency" not only the speed of a single query, but a property of the whole application (performance, stability, mantainability, etc). If your microservice design is not better than your monolith design, definitely go for the monolith.

Answer 2

If you need a join on the data of two Microservices, your Microservices design is simply not correct. Data that logically works together in your system belongs in the same Microservice.

Another alternative is to duplicate the data of the user microservice into other Microservices as needed using events E.g. relations into posts (but not addresses) and addresses into billing (but not relations). Note that this way you can end up with a distributed (tm) big ball of mud (tm) pretty fast.

Answer 3

Define a view database, which is a read-only replica that is designed to support that query. The application keeps the replica up to date by subscribing to Domain events published by the service that own the data.

https://microservices.io/patterns/data/cqrs.html

Answer 4

If you blog service accepts a list of userIds as input, ie

select posts where userid in (1,2,3,4)

then there is no particular inefficiency.