2

Currently, I'm working with a health management application, let's call it application A. It's a partly prebuilt generic application that we are extending on. The main entities are journals, events, and persons(patients/medical staff/external contacts). The user can use the application through a React app to see medical journals with the most necessary data about a person's medical condition and historical medical-related events. It's also possible to create new events in the medical record. Several data grids show information about the leading entities in the applications, for example, name, age, address, event data, event category, journal name/no, etc.

In the same corporation, we also have an internal API that exposes person-related data to several applications. This API is a large data register containing data for several million persons, addresses, etc. Most of them are not related to application A since they don't have a medical journal and properly never will. This database can, on rare occasions, receive updates from national registers. This could, for example, be the address of a person that can change. Sometimes columns for all persons have also changed when data are represented in a new way. The API's responsibility is to expose the latest data related to a person for several applications.

Currently, what has been done is that all persons are being copied to application A from the API, even if they don't have a medical journal. Then several scheduled jobs fetch all new data every half hour which can be problematic since these can take a very long time to process, and it's just a pain to keep them in sync. Currently, there is also some confusion with which application is the authority of the person-data.

Now we have two options to mitigate the problem we are facing. We have come up with proposals.

The first proposal is to hold only an ID of the person, and every time we need to show a page that shows the medical condition, we have to call the API to get the name because the name is present on that page. We also have to develop solutions to joining data that users can use for the data grids. Since most tables use service side filtering with graphQL, this is problematic if we need data from multiple sources. If the API goes down, then application A will not work. The good thing about this solution is that the person-related data is only stored in one place.

The second solution is only to have person-data in application A that are connected to a medical journal. The data is then checked asynchronously for updates when the user tries to access personal information on specfic pagers. If there are new data, the user will get notified, and the updates will then be transfered and stored in application A's database. In this solution, the application can still work if the API is down for some reason. And the problems with the data grids are not a problem since all the relevant data that should be showed in a data grid are stored in application A's database. The downside is, though, that some data are copied.

As far as I have read, duplication in, for example, a microservice architecture can be wrong, and others say it can be a tool you can use to decouple the different services, for example, here Microsoft's microservice guide. I'm more into the second solution since I think the services are too coupled in the first solution, and the services can be too chatty, and even though it is the "same" entity, they live in different contexts. But I still have a feeling of unsureness.

Does anyone have any advice on how to deal with that? Or maybe is there another way to solve problems like this?

2 Answers 2

6

Generally having a single "source of truth" or "authoritative source" is the best solution. If you have a microservice architecture then holding the personId and calling the person Api to get that persons data is how the design is supposed to work.

Your reasons for rejecting this seem thin.

  • The services are too coupled.

    There is the same amount of coupling in either case. If you copy the data you just have a longer held cache.

  • They live in different contexts

    The person has a single context, the medical records have another.

  • The services can be too chatty

    Caching can reduce the chatter, but given the current setup you would naturally expect the application to treat the person data as its own, loading at will. Refactoring might be desirable.

  • If the person API goes down so will the medical app.

    I expect there are parts of the medical app which can fail independently too.

  • Calling multiple sources is hard

    Add a "backend for front end" to call both sources and combine the data to fit the view

The upsides to segregating the data seem huge. From your description it seems like your medical application would break some data protection laws, limiting the data storage to a single place will allow you to reduce the risks of data leaks as well as get rid of the awkward syncing of data.

I can imagine some scenarios where you would want to store the data, where you need offline functionality for example; Or perhaps you need to query People with condition X and Address Y.

But I would resist this temptation as long as there is some other work around. As a rough rule I would say that if the data needs to be synced with any updates, then having a single source of truth, with its own api is the way to go and it sounds like you use this API in other applications already.

3
  • Thanks for the answer - that are definitely some valid points I will take into consideration. One of the things we discussed is how to query on persons that have medical journals without the API is aware of which persons have journals, of how we can do server-side sorting of a list that consists of journal-info (from app A) and name info (from the API). How can I solve this without having an SQL view that combines the data?
    – XRaycat
    Commented Jul 12, 2021 at 5:39
  • random querys become harder, but if you have something common like people by area, then bring that id in as well so have personId and AddressId or something
    – Ewan
    Commented Jul 12, 2021 at 10:13
  • 2
    IMO this approach settles the road to the distributed monolith and clashes hard with the idea of bounded context. It's the easiest implementation though
    – George
    Commented Dec 15, 2021 at 19:22
3

Use Event Driven Microservices and duplicate data. Its the simplest, quickest, easiest and most robust way to do these things. Whenever you have to directly call a service from another one you need to think about why. That in itself introduces coupling and creates dependencies - the whole thing you're trying to avoid by using microservices.

The answer for me is simple, use events to propogate changes in your system. Store exactly what you need for each service to function completely independently and remove the direct calls to other API's. Each service should be useful on its own - thats how you leverage the benefits of the architecture.

4
  • in this case I think you would be duplicating the person service in all your other services.
    – Ewan
    Commented Dec 16, 2021 at 9:12
  • You duplicate the specific parts of the Person service that you need in the other services. A limited sub-set only - not the entire service. You only need the data parts. So for example. All person fields are stored in the Person Service, but in another service you might only store a PersonId and a PersonName - because all you need is to know that that person exists. If that makes sense? Commented Dec 16, 2021 at 9:23
  • yeah just in this case it looks like you would need the whole thing, so you would end up duplicating or just storing the id have making the call as required
    – Ewan
    Commented Dec 16, 2021 at 17:19
  • I like this approach, and I am pushing to investigate if Kafka is the right tool. Currently SSIS updates the source DB from an external source, and I can see that Kafka connecters can emit such kinds of update events - which is perfect. It's not all data that are duplicated. Maybe it's 90%, and it could be less than if the data are analyzed. Using data tables that can do complex filtering/sorting is essential, and currently, the business is not ready to give up this functionality.
    – XRaycat
    Commented Jan 11, 2022 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.