0

Its an infrastructure monitoring application.

We have a spring boot (2.x) based Service Oriented Architecture (SOA) with multiple modules in our project and the center point is a REST services module which exposes multiple APIs which are in turn used by a client (Angular based).

We are brainstorming over a module which will handle all the exports (excel, csv, pdf, etc.) requested by a user via the client.

There are export button at multiple dashboards on the UI. Typical examples would be:

  1. Dashboards where we show data in tabular format and there is server side pagination implemented in it.
  2. Dashboards where we show data plotted over chart and the user can request a "bulk download" for extended period of range which is not possible to plot and make sense over the UI

A few points about the REST services module must be kept in mind before moving further

  1. The same module can be deployed using 3 different spring profiles
  2. Profile change leads to different database configurations (profile 1 will connect to DBs of infra 1, profile 2 will connect to DBs of infra 2, and so on.)
  3. There are a few beans (entity, services, controller) which are only available in one profile and absent in another. We control this using @Profile annotation
  4. All the API requests are passed through a API gateway before getting into any of the REST service deployment. We are using openid connect for authentication and authorization

A typical deployment looks like this

enter image description here

Its the same code base being run using different profiles. It allows us to have a single code base to maintain, for all the infra types. The nature of infra is such that there are very slight differences in them (90% of services are exactly same) and we are able to do cater such things using @Profile annotation.

Now the idea behind having a standalone exporting module is to not impact the servers on which the REST services are deployed (as some exports are very CPU and memory hungry) and divert all such requests to a cluster of exporting modules which can connect to any of the underlying infra DBs. This is where the problem starts.

The exporting requests are CPU and memory hungry but not very frequent. So it would not make sense to have a dedicated exporting node/cluster per infra. It has to be multi-tenant (don't think of spring multi-tenancy now) in some way.

Let's take one example now.

We have a page on which we show approximately 50,000 records using server side pagination in tabular format.

The user might request to export the whole data in excel format. This data is not pre-cooked and we do multiple transformations on the data received from database to show in a tabular format. There are other steps also in the process.

Point being, there is a service layer with some transformation logic after the data is fetched from DB, which gets served to the client through a controller.

The user is expecting the whole data to be present in the excel he just requested in the exact format its being shown on the UI.

As we are going to delegate this task of fetching and creating an export file to a separate module, we need some code overlap (the service layer and the repository layer).

As the module can connect to only 1 infra at a time, including the services and repositories in the exporting module will only allow it to connect to one infra at a time. This means, if there are 3 infra, we need 3 separate exporting modules as well, which is an overkill, as the requests are too infrequent.

We have three approaches in mind to solve this

  1. Make the REST service module itself multi-tenant so that every module can connect to every infra, based on some parameter. If we do so, we will not be able to maintain the non-intersecting features using @Profile annotation as the REST services module now becomes a super-set of all features.
  2. Include the exporting logic in the REST services module itself and add 1 more profile per infra which will disable all other features and just run the module as a exporting server with relevant beans available in the spring context. I'm not sure whats technically wrong in this approach but it just doesn't sound a good thing to do. Also, there will be 1 separate node for every infra in this case as well.
  3. Made the exporting module behave as a client of the REST modules. Make calls over HTTP from exporting module to REST services module and then write the logic to write down data into the excel/csv/pdf. This approach looks the standard way of going forward but in this case we will hamper the REST services with repeated load as the processing will still happen on the server hosting REST services, just the processing to write data will be delegated.

Need your valuable suggestions on this problem. Let me know if you need more info. Even a single line could instigate us to look in the right direction.

0

I would like to join your brainstorming based on your requirement:

Now the idea behind having a standalone exporting module is to not impact the servers on which the REST services are deployed (as some exports are very CPU and memory hungry) and divert all such requests to a cluster of exporting modules which can connect to any of the underlying infra DBs.

1st solution make your every module can connect every infra but this does not solve your concern, right? It could be even worst, report requests made to your infra2 will make slower your infra1 services for no reason. (If I understand all your flows correctly)

2nd solution solves the different infra's blocking problem each other which comes with the first solution. But now there is one different export instance for each infra which is again what you did not want in first place because of high resource consumption for low demand.

3rd solution comes with a different cost. Now your source code is separated for different export module which means your DTO's should be also copied into this module. Also moving high volume of data multiple times (db -> infra -> export -> ui) instead of (db -> infra -> ui) comes with cost.

So, how about using 1st and 3rd option together? If you can provide your current solution with multi-tenancy enabled + as a separate module, you don't have to move your data multiple times in the network and you can separate the load between export and REST calls. Like the following structure;

UI ---/export-> Gateway -----> report ------> DB-infra1/DB-infra2 ----> UI

UI ---/get----> Gateway -----> infra1 ------> DB-infra1 ----> UI

UI ---/get----> Gateway -----> infra2 ------> DB-infra2 ----> UI

So report service is single (or can be scaled up) service that handles all infra types while you can still keep your current infra based @Profiles to separate the loads.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.