0

Overall situations:

The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into small MISes (something around 15), each of which with perfectly divided business specifications and roles. Each MIS works fine on its own, however since they are a module of a larger system, they need to be integrated and some data sharing between them should happen.

Following is the illustrations of how the actual system has laid down: Sample system architecture

Whereas:

  • MIS1, MIS2, etc. are the MIS modules which over all comprises the system. Totally there are about 15 MISes.
  • Masters is a system which holds the data which should be shared between all the systems and mostly comes in drop downs. It is also an MIS.
  • Accounts + sessions, is a system where all the users are created in and all the sessions in all the MISes are managed by. This enables the single sign on mechanism.

All these modules are connected between each other by local HTTP APIs which cannot be accessed from public.

Each MIS is a Sing Page Application.

We are using RBMS as main database.

Most of masters data are cached in browser local storage for the first time an MIS is loaded, and they will compliment the server data (which only has the slug from the Masters data). But in backend each time the Masters backend is called and we do not have a cache mechanism there.

Benefits

  1. It hides the complexity of each module. The team working on one module does not need to know the details of the other module.
  2. It supports the horizontal scaling, i.e. duplicating the same server can reduce the overall load of one module.

Problems and challenges:

When we followed this architecture, we faced the following issues:

  1. Since there are a lot of HTTP requests back and forth between the systems, some requests take a long time to respond, e.g. in reports it might take around 30 sec to process.
  2. Sometimes we need the data from the Masters database and Accounts database to be used inside a database query, which we do not have a clear strategy how to handle them.
  3. HTTP request between the local servers seems overkill to me.

How do we deal with the problems we are facing?

Update:

The following gives some feeling on how the data trasfer might look like (however our use case is different): Sample data relations between MISes

In this scenario, I have a employee management MIS, which takes data from masters and learning MIS. We have integrate the employees database with provinces table of masters database in the following manners:

  • All the details of all the provinces of Afghanistan are entered in the provinces table of masters database, one of its field being the slug, which is unique name derived from name_en (English name of the province).
  • slug is then used in the other databases where the province needed to be referenced. The relation is rather logical than having FK.
  • Now, if in the back end we need the other details of province, we send an HTTP request to Masters back end to get that. The response of the request is merged with the query of the employees database and the overall result will be sent to client at once.
  • If there are pieces of data in other MISes, the same procedure might happen for them too.

The above set up will be time consuming when there are many fields to be fetched from other MISes.

Another problem will be the aggregations. If we need to aggregate the data based on a field which is not in the current MIS database, we need to keep a duplicate data of the same data set in this database too, which the synchronization is a problem with.

HTTP seems an overkill for inter-MIS calls to me since HTTP checks so much for fault tolerance, which is time consuming and I do not think we need, since the network has a stable condition, however it might be only my own assumption.

30 sec is a pretty long time for landing pages, for example in learning MIS, the batches list page would be a headache for all the users.

  • 1. What sort of information is being communicated between MIS's? Why is it taking 30 seconds for reports to process? Does 30 seconds really matter for a report? – Robert Harvey Jun 27 at 15:02
  • 2. What does "to be used inside a database query" mean, and why is this a problem? – Robert Harvey Jun 27 at 15:04
  • 3. "HTTP request between the local servers seems overkill to me" is an interesting observation, but what are the practical implications? – Robert Harvey Jun 27 at 15:04
  • @RobertHarvey thank you for the attention and questions. I have updated the questions to reflect your questions. – Musa Jun 29 at 4:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.