Querying data from multiple microservices

Question

I'm trying to design an application that uses the microservices architecture. The service would have an api gateway, a user service and then a seperate service for each analytic source.

For example, lets say I had twitter analytics data and instagram analytics data. There would be a seperate microservice for twitter analytics data and instagram analytics data. These microservices would handle oauth for that service, storing data and pulling data. My question is this: how would I request data from one service, lets say the user service, and then based on data from the user service, make a request to the instagram service and twitter service to request analytics info for that user.

However what would happen if I had n amount of services that I needed to request to? Like let's say a user needs analytics data from facebook, instagram, twitter and tiktok. How would I handle making a seperate request to each and then combine into a single response for the client?

I was thinking of using an api gateway as I've read that you can support request fanning-out to multiple microservices. I'm exactly sure how to do that however I found this article from aws outlining the different strategies.

I've also read over this question that says "You should implement endpoints in the services to facilitate these two requests. If you're making n requests, you've done it wrong". So now I'm not even sure if I'm on the right track.

Am I approaching this the correct way?

Answer 1

lets say I had twitter analytics data and instagram analytics data. There would be a seperate microservice for twitter analytics data and instagram analytics data.

I feel there is a need to seperate the services as each of these services are fetching a users analytics from a third party api (twitter, facebook, etc) and then storing the analytics. If all the services were combined into 1 monolith, wouldn't it be hard to add more third party providers and manage everything in general?

There's no one true answer. But instinctively I'd be considering the position of having a single "analytics service" rather than a per-external-party service.

The issue is that the per-external-party service approach means that downstream consumers are affected by the addition/removal of a third party, which means you've got a leaky abstraction on your hands. Your consumers need to know which third parties are available so that they can call the appropriate service, which defeats the purpose of abstracting that dependency away in a separate microservice.

By keeping it together as "the analytics service", you obfuscate which third parties you rely on, and you can add/remove third parties by solely redeploying "the analytics service" without impacting your downstream consumers.

But like I said, there's no one true answer. It depends on what you need. I'm mentioning the approach that you've dismissed simply to point out that it can be equally valid.

However what would happen if I had n amount of services that I needed to request to? Like let's say a user needs analytics data from facebook, instagram, twitter and tiktok. How would I handle making a seperate request to each and then combine into a single response for the client?

The bolded part especially leads you down the BFF (backend for frontend) pattern, although the consumer here might be a different backend service instead of a frontend.

The focus here is on the "General Purpose server-side API", which is effectively a microservice that fetches data from several other sources (the ghostly icons at the bottoms), collating it into a single data response, which it then sends on to the downstream consumer (commonly the frontend).

Additionally, we can considered that this intermediary service is really just "the analytics service" which I talked about before. It presents the analytics as a single front, to hide the complexity of having multiple third party providers.

The same question then arises: do you really need those additional separate per-external-party services? The answer to that is found by figuring out what it nets you.
For example, if you need vastly different scaling for these different third parties, then it can make sense to have these as different runtimes/services. But in absence of such a need, I would argue that having a single analytics service makes the most sense.

Note also that once you have this analytics service, nothing is stopping you from later adding those separate per-external-party services. Adding them will not impact the downstream consumer of the analytics service, since the analytics service breaks any direct dependency between a downstream consumer and a specific third party.

Answer 2

A sequential description would be (1) one request to retrieve some state followed by (2) n requests to retrieve analytics based on the passed in state followed by (3) the aggregation of the information retrieved by the n requests followed by (4) a notification sent to the service initiating the request for aggregated analytics.

The transition from (1) to (2) is straight forward it can be implemented following observer design pattern, have a singleton observable service retrieving a state notifying its observers once the state is consistent. The transition from (2) to (3) is the reverse of observer design pattern, multiple observable services notifying one observer, the aggregator service, that uses builder design pattern to aggregate data. The transition from (3) to (4) is observer design pattern with the builder notifying the its observer with the aggregated data.

The tricky piece is to trigger the data aggregation. It has to support service failure and aggregate partial analytics in case of non-responsive analytics services. Since I don't have knowledge of a suitable design pattern, let's improvise and implement an observable service that supports instance per call and that guarantees notifying its observer regardless of the result of the call to the external service. Then have created an instance of early described service for each analytics to be retrieved with the aggregator service instance observing it. Then the aggregator service instance has to have a stack of no-op of size n - 1 so the nth notification triggers the aggregation or has to support a state diagram and with each notification advance the state until the aggregation, with the side note that the aggregator service also has to support instance per call.

Now some hazardous advertising, maybe Spring Webflux is what you are searching for.

Answer 3

Each analytic service can be its own service (n total of them) so that micro service architecture is achieved, but at a higher level there would be an aggregator service which would be responsible for taking the data and returning a single result.

Internally, the aggregator could have some sort of orchestrator to get the data. Typically, it would manage all the async callouts to different services. It could do this via async chaining and/or awaiting the return of multiple service calls, before aggregating the result.

Pseudo code with calling 3 services, waiting on the first:

// Wait until first service is done
var task1 = DoFirst();
await task1;

var task2 = GetTwitterData(task1.result);
var task3 = GetInstragramData(task1.result);

await Task.WhenAll(task2, task3);

return aggregateResult(task2.result, task3.result);

One could separate the aggregation and orchestration if needed. But now this is a single endpoint to call as the aggregator handles internals of aggregation.

Answer 4

A design approach that be considered for this is to use CQRS. A query database tailored for your query can be useful. Data replication can be handled in different ways - through message brokers, CDC, event bus or dedicated services etc.

Few drawback of this approach

1. Data consistency - again it depends on how the consistency is maintained between the write or master source and the query source.
2. Cost of maintaining additional DBs and services.