2

We have a system with a few microservices and multiple clients (Web, mobile). Currently, only one microservice (let's call it "Master MS") exposes a public API, which is used by all clients. This worked well until now: Master MS did occasionally ask for data from other microservices, aggregated it, and sent to clients. Now the system started to grow.

PROBLEM 1:

We have now more microservices that should provide data to end clients on the one hand, and Master MS should not really know that these microservices events exist.

PROBLEM 2:

Client has very specific needs now (Web client wants more data in coarse requests than mobiles, also some pre-evaluated data, e.g. not GET /resources1 and GET /resources2 but GET /info {hasResources1 = true, hasResources2 = false}).

So we want to go with an Application Gateway (or more like Backends for frontends) pattern and currently, YARP is our favourite for the task (lots of .net developers in the team, low cost as open source). The idea is to set up two YARP gateways, for web and mobiles, that reroute to backend microservices. PROBLEM 1 is thus solved, partly also PROBLEM 2, and here comes the question.

Where should I do the aggregation? Let's say Web client wants to get data for the dashboard. This involves e.g. asking Service1 for customer data and sending two requests to Service2, for let's say purchase and client history. All these responses should be transformed into a single DTO.

So who hosts this aggregated endpoint? YARP gateway itself? That seems to be not the purpose of YARP, I cannot find a place or way to put the code there in a meaningful way. The code also doesn't fit either to Service 1 nor Service 2. So where should it go?

Improve this question
3
  • 1
    Why not an standalone service?
    – Laiv
    Commented Feb 14, 2023 at 21:19
  • @Laiv that kind of defeats the whole point of using a ready made API Gateway and implementing it ourselves, whereas the aforementioned functionality is needed for like 10 endpoint, and all others are just simple pass-throughs (i.e. a gateway fit perfectly).
    – Maxim Zabolotskikh
    Commented Feb 15, 2023 at 7:22
  • Well, some may say that aggregation is bringing business responsibilities to the API Gateway. if you consider an API Gateway an infrastructure element (implementation detail that should be possible to replace without impact to the business) then the standalone service doesn't defeat the purpose of the API Gateway. Some Microservices can be composed of smaller Microservices leaving one as a facade. An aggregation service can be easily implemented with programmable ETLs such as NodeRed (to mention one)
    – Laiv
    Commented Feb 15, 2023 at 11:34

1 Answer 1

Reset to default
4

In a microservices architecture, the responsibility of data aggregation should ideally be with a dedicated service that is responsible for aggregating data from different microservices or the client application.

In your case, since you have different requirements for the web and mobile clients, you can create two separate services responsible for aggregating data for each of the clients. If you go with this option, the YARP gateway can route the requests from the clients to the appropriate microservices based on the request path and headers. Once the microservices have responded, the responses can be forwarded to the respective data aggregation services, which can aggregate the data into a single DTO as required by the client.

Alternatively, you can also consider implementing the aggregation logic in the client application itself, using a client-side aggregation pattern. This can be achieved by making multiple requests to the microservices from the client application and aggregating the responses within the application. This approach can provide more flexibility and control to the client, but it may also require more development effort and complexity.

Improve this answer
5
  • Do you know if it's possible to map a single request in YARP to several target paths and aggregate the responses? Like with some transformations? I cannot really find an extension point there. What works is putting an aggregation service behind YARP, but than that service will have to call microservices by itself, and many benefits of YARP, like load balancing, are lost.
    – Maxim Zabolotskikh
    Commented Feb 20, 2023 at 14:44
  • Yes, that's possible. Take a look here: microsoft.github.io/reverse-proxy/articles/transforms.html
    – Sebastian Sotto M.
    Commented Feb 20, 2023 at 23:29
  • I had seen that section - it allows to do pretty everything with the request, but it does not allow to take one client request, send it to >=2 destinations, wait for the results and aggregate them in a single response. That's basically what I was after..
    – Maxim Zabolotskikh
    Commented Feb 21, 2023 at 7:44
  • @SebastianSottoM. how do you implement aggregation as a dedicated service, actually? do you have some example? tnx
    – tlt
    Commented Dec 8, 2023 at 1:35
  • Yeah sure. Consider an aggregation service designed to unify data retrieval for an application in the retail sector. This service exposes an endpoint, say /mobile-data, which internally communicates with some microservices. The service may utilize HTTP requests or asynchronous messaging to fetch product availability from an inventory microservice, customer details from a customer microservice, and order information from an order processing microservice. The aggregation logic then combines these data sets, perhaps using JSON payloads, into a cohesive API response for the mobile client.
    – Sebastian Sotto M.
    Commented Dec 15, 2023 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.