The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into small MISes (something around 15), each of which with perfectly divided business specifications and roles. Each MIS works fine on its own, however since they are a module of a larger system, they need to be integrated and some data sharing between them should happen.
- MIS1, MIS2, etc. are the MIS modules which over all comprises the system. Totally there are about 15 MISes.
- Masters is a system which holds the data which should be shared between all the systems and mostly comes in drop downs. It is also an MIS.
- Accounts + sessions, is a system where all the users are created in and all the sessions in all the MISes are managed by. This enables the single sign on mechanism.
All these modules are connected between each other by local HTTP APIs which cannot be accessed from public.
Each MIS is a Sing Page Application.
We are using RBMS as main database.
Most of masters data are cached in browser local storage for the first time an MIS is loaded, and they will compliment the server data (which only has the slug from the Masters data). But in backend each time the Masters backend is called and we do not have a cache mechanism there.
- It hides the complexity of each module. The team working on one module does not need to know the details of the other module.
- It supports the horizontal scaling, i.e. duplicating the same server can reduce the overall load of one module.
Problems and challenges:
When we followed this architecture, we faced the following issues:
- Since there are a lot of HTTP requests back and forth between the systems, some requests take a long time to respond, e.g. in reports it might take around 30 sec to process.
- Sometimes we need the data from the Masters database and Accounts database to be used inside a database query, which we do not have a clear strategy how to handle them.
- HTTP request between the local servers seems overkill to me.
How do we deal with the problems we are facing?
In this scenario, I have a employee management MIS, which takes data from masters and learning MIS. We have integrate the
employees database with
provinces table of
masters database in the following manners:
- All the details of all the provinces of Afghanistan are entered in the
mastersdatabase, one of its field being the
slug, which is unique name derived from
name_en(English name of the province).
slugis then used in the other databases where the province needed to be referenced. The relation is rather logical than having FK.
- Now, if in the back end we need the other details of province, we send an HTTP request to Masters back end to get that. The response of the request is merged with the query of the
employeesdatabase and the overall result will be sent to client at once.
- If there are pieces of data in other MISes, the same procedure might happen for them too.
The above set up will be time consuming when there are many fields to be fetched from other MISes.
Another problem will be the aggregations. If we need to aggregate the data based on a field which is not in the current MIS database, we need to keep a duplicate data of the same data set in this database too, which the synchronization is a problem with.
HTTP seems an overkill for inter-MIS calls to me since HTTP checks so much for fault tolerance, which is time consuming and I do not think we need, since the network has a stable condition, however it might be only my own assumption.
30 sec is a pretty long time for landing pages, for example in learning MIS, the batches list page would be a headache for all the users.