0

Overview

I've got 2 systems over a network. The 2nd system produces some results based on the current contents of system A. How can I efficiently produce results in system B if getting the contents of system A takes a long time?

Current sitation

  • I've got a system, system A, that keeps customers (personal details, etc). The user can manually enter hits in this system.

  • I've got another system, system B, that performs, every night, some checks on each customer. It checks if a customer appears on a restricted-persons list producing a report of hits. A user checks on this system each morning and screens manually the produced hits. If a hit is approved, it is then send via POST for saving into system A.

The problem

User might have entered a manual hit in System A, which was also found automatically in System B.

Solution A

Send a GET request to system A, every time I present system's B overnight results to the user, to get the existing hits and filter them out of hits produced by System B

In order to ameliorate this mismatch, every time I present the results of system B in the morning, I can send a GET request to get the existing hits of a customer in System A and then perform a comparison, filtering out hits that exist in A and B's hits.

This allows me to show only the new hits in System B. The user screens only the new hits that were found.

A problem with this solution

I can't efficiently know beforehand how many actual new hits exist in System B. I need to GET 50,000 customers, from System A, each time I want to get the number of new hits.

Solution B

Send a GET request to system A, when a hit is produced by System B, overnight, to see if it's really a new hit that doesn't exist in System A.

Problem with this solution

Since the GET request is sent at the time a hit is produced while checking the list and not when the results are presented to the user there might be a mismatch. The user might have not tended to the results of System B and went ahead and manually entered a hit in System A. System B would still show that manually-entered hit from System A as a new hit in System's B morning results.

Is there a standard way of dealing with such problems?

Details

  • The 2 systems are built on Node.js/Express framework. I have build those systems myself so I have full control over them.
  • The systems can communicate with each other using a REST API.
  • I'd like to keep the 2 systems decoupled, honouring the Single Responsibility Principle - i.e System A just holds it's data, System B performs automatic screenings etc. I'd specifically like to avoid intertwining functionality between each system
  • Why can't system A just check to see if the hit from system B was produced manually in A? If true, ignore the hit from B. – Mike Mar 30 '17 at 11:03
  • @Mike The morning results of the search are presented in system B - System A only receives via POST from System A. – Nik Kyriakides Mar 30 '17 at 11:08
  • So what is the impact if the user of B approves a hit, but unknown to the user it was already manually entered into A? – Mike Mar 30 '17 at 12:15
  • @Mike Not a big impact whatsoever, apart from the user having to screen a hit in B that was entered thus already screened in A. – Nik Kyriakides Mar 30 '17 at 12:16
  • @nicholas-kyriakidesn Then maybe this is a non issue and you needn't spend time on it? – Mike Mar 30 '17 at 12:42
0

I've solved this with a combination of the following:

HTTP Caching

To get the new hits, I send a GET request to grab the hits from system A. That GET request is cached for exactly 1 minute. Any subsequent requests are served from the cache instead of fetching over the network again.

There might be a mismatch in the new hit count but it's for very brief periods of time (when the cache is stale). I can certainly live with that

Bypass cache when user focuses attention and get a fresh copy

When the user focuses on a particular hit (and is ready to transfer it), I perform a direct GET request again for that particular piece of data, bypassing the cache.

This ensures that when user is actually about to make a decision/operation on a hit, he is presented with the guaranteed, non-stale piece of data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.