Should queries in CQRS be designed based on external demand?

Question

I'm working in a microservice application which has some level of coupling between the microservices (some of them talk with each other via rest api - I know it is bad, but it is what it is; also I think this question would still be the same if we changed this microservice intercommunication problem with the normal front-end application use-cases/needs).

Each microservice application was implemented using CQRS and following DDD (or at least trying to).

I'm implementing a specific feature where one microservice needs to group a lot of data that is owned by a few other microservices and return this grouped data to the user via its rest api. I can think of two possible approaches to that: A) queries (CQRS) are designed to be generic and reusable or B) those queries are designed with a specific external demand in mind, thus not being very reusable and in practice having a 1:1 relation with a specific use case (ie. 1 query per use-case).

If I go with the first approach, I'll have a code that is easier to maintain and read (those that are related to the queries), but eventually there will be some use-cases that will not be answered in the best performant way. However if I pick the latter, I'll have potentially a lot of queries created to answer other microservices needs (possibly evidencing knowledge leakage between those microservices) that will be unfeasible to even know if that query is not being used anymore, those queries will probably be forever occupying LoC in those projects.

Reading about CQRS I think the correct approach would be to have 1 query per use-case as CQRS is used mainly to solve scalability problems, hence it does make sense to have a query created to answer a specific use-case in the best possible way, but I can't stop thinking this would violate some fundamental principles (like encapsulation for instance). Is my thinking right? What would be the correct answer here?

Answer 1

What you are dealing with here is the age-old question of performance (through tailoring) versus maintainability (through generalization). Spoiler alert: there is no right and wrong answer here.

This problem is entirely unrelated to microservices; as this same quandary can be found in considering the design of a DAL of any given codebase.

If you stick to generally reusable logic, you are able to cut down the amount of code you have to maintain, while also avoiding the potential mistake of muddying the separation that should exist between your layers (or in this case microservices). You want to avoid having to develop something in A just because B needs it, because that only makes sense in a combined AB project, not with independent A and B projects.

But on the other hand, doing it this way means that you may lose out on significant query optimizations that A's database has to offer. This might be a problem for you. This is where you have to weigh your options. Would you rather squeeze for performance, or minimize the maintenance effort on your codebase? While it's generally nice to minimize maintenance effort, there may be business considerations why the squeezed performance is worth such a cost.

This is a business decision, and not one strangers on the internet can make for you.

One client I worked for took a consistent "the API decides how it serves its data" stance, specifically avoiding A ever implementing something because B needed it. The reason for this choice is that the company had many customers who all spoke to one another, and if they did something for B, they would get C, D, E, ... to all ask to do the same thing for them as well. This shifted the dynamic to custom-tailoring code for each individual customer, which the company did not want.
Even when presented with clear evidence of mediocre performance on certain implementations, the company stuck to their guns. The CTO maintained that trying to tailor code to a specific use case was a slippery slope, and due to the sheer complexity of the business domain, it was a slope he did not want to slip on, not even a bit.

Another company I worked for really prided itself on its customization and tailoring to fit a client's needs. They would happily customize and tweak any part of the product if it would get them a new client or additional work from the same client. While they did try to keep things reusable behind the scenes, when the choice came between reusability and custom-tailoring, they chose custom-tailoring every single time. While this added to the maintenance effort of the codebase(s), the addded business and revenue it gained them was worth the extra cost of development.

There is no right answer here. You have to consider your business, and how these decisions impact the quality of your final product and the development cost of making and maintaining the product.

---

The rest of this answer consists of feedback on things you've stated, because I feel that while you are mostly in the right ballpark, you've made some inaccurate statements which are influencing your overall decision making process here and can lead you to make the wrong choice due to a subtle misunderstanding.

---

some of them talk with each other via rest api - I know it is bad, but it is what it is

This is not bad at all. Microservices are perfectly allowed to talk to one another. What matters is that they have independent lifecycles, but that doesn't mean that one microservice cannot contact another.

A) queries (CQRS) are designed to be generic and reusable or B) those queries are designed with a specific external demand in mind, thus not being very reusable and in practice having a 1:1 relation with a specific use case (ie. 1 query per use-case).

In general, even when an application largely consists of simple and reusable CRUD operations, you tend to always have a few queries (or commands) that are so niche that they only serve one use case. There's nothing wrong with that, and in my experience it's almost impossible to avoid in any codebase that is not a trivial data storage service.

That being said, if you take the deliberate stance that the provided service should be independent and not made to supply specific needs of specific consumers, then that's fine too.

but eventually there will be some use-cases that will not be answered in the best performant way.

Microservices are not built for maximized performance. All other things being equal and considering a single request (as opposed to serving many requests at the same time), monoliths outperform microservices simply by having a single runtime which facilitates all necessary steps of the request. Microservices have to communicate between one another, which comes with an overhead cost.

That doesn't mean that you should totally ignore performance when developing microservices, but just don't expect perfectly optimized performance to come from the microservice pattern.

You are correct that scalability is one of the reasons for choosing microservices. Not that you can't make a scalable monolith, but being able to selectively scale one microservice and not the others can yield greater efficiency in terms of required hardware, server upkeep and maintenance.

Answer 2

I think you're in good direction considering the CQRS path. In such scenarios where the UI requires views composed of data from several of your microservices, a good approach is to build a view model per use case. The alternative will be continuing on the path of the distributed monolith you seem to be working on.

In order to keep them sync you'll probably have to embrace eventual consistency and update the models through a message broker; otherwise you will be coupling the application command and read sides. In ES projects you typically subscribe your read model store to the event store's stream of events and process them in an ordered way. Note that this adds a lot of complexity - eg: errors that leaves the read models in an inconsistent state, assuring that the view models are updated in such a way that the system is causal, etc -

Answer 3

Your decision shouldn't depend on the architecture at all. The degree of reusability of your interface and query implementation (both of which you should be considering, since you are designing user-facing code) should depend mainly on whether or not (and how often / how much) you intend to reuse this code.

How likely is it that the problem that your api / query will be generalized and / or expanded in the future? You can start by considering the following points, all of which should share a strong correlation with your decision.

• can the problem you are solving be generalized? If you're writing a program to price bicycles in a POS system, and this system belongsto Walmart, you should be writing in a way where The bycicle element is arbitrary. If it belongs to bycicles-only.com, on the other hand, you will probably not need to worry about generalizing. This example is intentionally obvious, but this isn't always the case.
• How close is your current feature to the web-app's core feature-set? Development team will be much more likely to double back over, and might even wear a path through, territory that is directly adjascent to "their front yard." You know the tendencies of your coworkers better than anyone here. I would imagine that you can probably get a rough estimate as to whether or not this feature will be reused .

One final consideration is a bit less obvious. That is the age of the project. As a general rule, (and general is the keyword here) the more a project matures, the less you will need to focus on reusability. Although it is always a great habit and a best practice, it is especially important when building out the framework and infrastructure of a project.

For example, one major concern that you are getting ready to face is dealing with / working around the updating of the query-side data store. As it is read-only, you don't need to aim at a moving target as usual, but when it does move (when syncing with the 'command' side of the system,) it usually involves a much bigger change. In other words, if this is ignored, your query could finish in an environment that is completely different than the one in which it started.

As you probably know, this is usually best dealt with via some sort of library command or helper api. Maybe a daemon that keeps time with the update (and maybe even handles them) and sends an "okay to cross" event. The implementation doesn't really matter here. The point is that problems like this recurr frequently, so solving them in a robust manner early on (ideally at square one) can save a project countless man-hours in the long run.

If your system is sufficiently mature, these will already be included in a sufficiently robust form. If they are not, then your project is likely still in the "settling" phase. In this case it is on you to determine the difference between the infrastructure and the dressings, and to make sure that the former is sufficiently robust for whomever may need to use it next.

Answer 4

IMHO, I think that your question is not much related to the CQRS Pattern. It's focused more toward the boundary of the services.

We are all agreed that the microservices pattern lets each service operate independently. When you create a composite query that collects data from many services, if one service fails, the query will fails, meaning all fail, and the pattern broke. Therefore, your B option seems to be anti pattern.

I would choose to implement the A option. Based on client demand, there are 2 options:

• If the client needs fresh data, (e.g. product and real-time auction bidding) then the composite query will be placed in front-end, then it will collect data by querying from related APIs.
• If client doesn't need fresh data, (e.g. produce and inventory stocking that can be updated daily) then you can implement a cached table, against which you can run queries. This table will be updated periodically, using the Event Sourcing pattern.