1

Suppose we have multiple services which each of them uses its own db(SQL or NoSQL, could be physically distributed). How can handle queries which require data from two or more microservices?

PS 1

As an example suppose Users (id, first_name, last_name,...) and Orders (id, user_id, goods_id,order_date,...) are two micro-services in our problem. We want orders which first name of buyer is john and order date is less than one month. Suppose these two services are distributed across multiple servers and database of one them is Relational and the other is NoSQL. It seems we must first fetch all users with john as first name then pass user_ids to Orders and filter orders due to user_id and order_date(or vice versa could be possible in other cases, e.g. fetch records from service B and filter with service A is also meaningful). So if count of users with name john is huge or count is normal but request rate is high this could be problematic. How can we handle this?

PS 2

Also in comparison with a monolith app with single db which has these tables, query joins two tables and probably query optimizer plane which compute answer first by filter users then by time or vise versa. In a micro-service architecture we loss this also. How can overcome this in micro-service architecture?

2 Answers 2

4

It's often difficult to talk about hypothetical situations and talk about them being problematic without hard numbers. There are so many different factors to consider here. Mainly, we don't have any idea what kind of response time is required. For example, you could have a query that responds very quickly but the results take a second each to process. In such a situation, optimizing around the initial query is rather pointless as producing more than one result per second will have no impact on the overall runtime of the process. This is somewhat overly simplistic but you get the idea. How long does it take to retrieve the users? Unknown. How long does it take to retrieve their orders? Unknown. How long would it take to retrieve the same information from a relational DB? Unknown. There are too many unknowns to have a specific answer.

Also, it's a much different question to ask what you do if you are actually in this situation or if you are trying to decide whether microservices are a good idea. I get the idea that this is more of the latter or maybe you are just pondering about this abstractly.

Microservices are not magical fairy dust that makes everything better easier and faster. On a technical level, they create a number of challenges such as this. They also have advantages. Most of the advantages are non-technical. In large corporations, trying to manage one giant database for all the teams in the company is infeasible. All the coordination and testing required to do that properly will add months to delivering changes. Microservices allow teams to manage their own DB, use the tools that suit them and create clear boundaries for how other teams interact with their systems. In other words, one of the most significant advantages is organizational and has almost nothing to do with creating the most optimal technical solution.

A much less common reason for using microservices is a monolithic relational DB scales vertically. As the scope of the data and the number of clients increases, the costs rise at an increasing rate and at some point, the maximum capacity of the DB technology can be exceeded. Microservices, (and nonSQL DBs) are much easier to scale horizontally.

The point here is that if you are going down the microservices road without one of theses issues or some other reason that forces you to do that, you are probably creating more problems than you are solving. There are many teams that went with a microservices strategy because that's what the cool kids were doing and now have a mess on their hands.

Now if you are in a situation where you have this kind of design and you know that the basic solution (the approach you describe in section PS1) isn't going to work, there are a few things you can do. The most performant is likely to be adding extra data (user name, in this case) to the orders DB. Of course, this takes development time and you'd need to know what the queries are ahead of time. This also has a storage cost as well. You could also create standalone indexes (perhaps as another microservice) that allow you to get to the orders you need more quickly. Another option is to use horizontal scaling process the query in parallel against multiple DBs. This is not an exhaustive list but should give you some ideas around possible approaches.

4
  • 1
    For clarification I must tell u I know Micro-services pros/cons and the Q is focused on one of them which I could not read about it elsewhere. Q: In order to satisfy query condition we must fetch all/most records of one or more services and pass it to next, since in monolith it does not need to fetch all/most and it can do it in a integrated way. last paragraph of your A is helpful. Thanks
    – Bonje Fir
    Aug 4, 2021 at 7:14
  • Another eg. Suppose an international SocialNet app with users(SQL db) and posts(NoSQL db) services. Query could be all users live in 'US' and has 'Comment''s in last month. Count of users live in 'US' and number of 'Comment's in last month could be exclusively huge but there is few users with both of conditions. In monolith db this handle effectively but I do not know what can do in a micro-service architecture.
    – Bonje Fir
    Aug 4, 2021 at 7:43
  • @BonjeFir OK, I tried to cover both bases because I wasn't sure what kind of question this was. No offense intended. For the specific case you mention, it's hard to give an answer without understanding the design of the microservices. But if we assume the users country and their comments are in separate datastores, I'd probably pull the users with comments in the last month and then lookup each user's country and store that.
    – JimmyJames
    Aug 4, 2021 at 13:36
  • @BonjeFir I think you are looking for some sort of general ad hoc query solution across DBs. While you might be able to find a technology that does this on top of separate microservices (perhaps e.g. with GraphQL) but there's no way to side-step the challenges of distributed systems and get the performance achievable when all the data resides in one system.
    – JimmyJames
    Aug 4, 2021 at 13:44
3

Such queries are just not well suited for microservices.

The point of microservices is that they should be loosely coupled. If you introduce queries between the services you end up with tight coupling somewhere in your architecture. There are also consistency and performance to consider.

This article describes a bunch of different alternatives, with advantages and disadvantages. A short summary of the alternatives from the article:

  • Selective data replication
  • Direct queries between micro services
  • Composite service layer
  • Joining data in the user interface
  • Views between database schemas

My personal view is that while micro services are incredibly useful in some cases, there are significant disadvantages. And these disadvantages are not always given the consideration they deserve.

3
  • Thanks. In the shared link author survey the data sharing patterns. Here I have a specific problem with one of those approaches. I choose db separation so after that I meet queries which has conditions such that each of conditions exclusively are in one service and could not filter many records by self...(also other comments may be useful). What can we do?
    – Bonje Fir
    Aug 4, 2021 at 10:11
  • @Bonje Fir Well, if databases are separate you query one database/service with one condition, take the result and use that to query the other database/service. This will have a performance impact, if that is unacceptable, pick another solution.
    – JonasH
    Aug 4, 2021 at 10:24
  • As I mentioned above I am interested in to find a way to solve this problem. On the other hand before changing the strategy is there any technique/idea to overcome with the challenge? or any argu to proof this approach(arch) was an anti-pattern for the problem.
    – Bonje Fir
    Aug 4, 2021 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.