Best practice for accessing data from an external legacy ERP system

Question

I have an architectural problem, and would be glad to receive some consultancy :). The application I am developing (let's call it system A) is responsible for management of work groups, where these groups have a leader, a set of members, roles, categories etc.

The information about the people who will become the group members come from another application (system B), which is legacy, was developed by an external company, and will not be replaced any time soon. This system maintains records of every employee in my company (about 300,000+), and is one day per month very busy processing the payment bill of these employees. System A can obtain the employee information through SOAP Web Services developed by System B, but these web services are slow, and not very scalable.

The problem is, some operations in system A (like querying which employees do not belong to any group) do need to check every employee available in System B. Therefore, in order to avoid the performance bottlenecks related to directly obtaining these data from B's web services, I thought in the following solutions:

1. The slow operations would be performed in background by a parallel thread every two hours and the results would be cached. This reduces the number of times A would need to access B's web services, but every query performed on system A would be, in the worst case, two hours out of date.

2. One time per day, during the non-busy hours (i.e. midnight), a job would query system B for employee data and store on system A database. This would avoid querying B's web services every time the same info was needed, and since the number of employees from B didn't need to be updated quickly, a single day delay wouldn't be a problem. The issue in this case is duplication of data on two systems.

3. Giving direct access to B's database to system A. This would eliminate any delay related to remote communication, and opens the doors for many database optimizations. The problem of this solution is high coupling between two systems (not very SOA-like) and operations on system A would increase the performance burden on system B.

I would appreciate opinions about these three solutions, if anyone experienced the same problem, and would be all ears for a 4th solution.

Answer 1

1 of course would most likely be the easiest option to implement.

2 is a reasonably common practice to help deal with slow or hard to deal with systems. And your right it does duplicate the data, so you would need to have the policy that System B is the authority and A should be used as 'read only'.

I personally would go for option 3, with slight modifications. I would replace the Web Service calls with an interface (dll) that goes directly to System B and have System A make calls to it. So still keeping the systems separate but giving you a better window into the system and more control.