Listing search/query results from 30+ web services efficiently and orderly

Question

We have a system that currently has approx 25 customers (will never expand from this) that each have a public facing web service which allows each other to query each others internal application information. e.g search application number (e.g PAR1056) or it's applicant name (e.g Bill Bob) and recieve a list of matching applications (e.g if i typed in PAR I would get PAR1, PAR2, PAR3 etc) and then once an application is selected, retrieve some information about it.

Currently each of these services have a basic front end written to query them in which the search field will go and call the services (automatically, not via post back) and list the results. It has become a requirement (sales team already sold dreams) to create a front-end portal that will allow any user from any of these sites to use a search box to type in the application number, or applicant name and retrieve the results from each of the 25 sites that are running the service (so basically a lookup service to query all the sites).

Having this sold out from under us, without a real concept already prepared, I am trying to consider how this can be done. There is no consistency between any of these customers other than the fact they are using the same service (i.e the system holding application information is 3rd party and different site to site) and geographically each of these sites are about within 1000 miles of each other. Each service contains anywhere between 10,000 to 100,000 applications so I think running a nightly sync service to bring all results to a central location is not a feasible option.

This places in my head the doubt of being able to query service A to see if it has results, if not move on to service B and so forth until the first 50 results are found. If there are no results until the last few services than the perceived performance is going to be terrible.

The only logical way I believe it can be done, is once texted is typed, all 25 services are called at once and it lists the first 50 results for each of these services in a nice front end that separates and distinguishes which site each of the results come from.

The other solution would be to call each of the 25 services at the same time to see if they have any result (Boolean true/false) and then run the listing service on only the services that do return true.

The aim obviously is perceived performance.

Has anyone done or worked on anything similar before, and what did they achieve with this?

Thanks,

Answer 1

Rather than centralising the date overnight, why not use a user interface, which naturally shows the progress on each of the sites being searched.

https://iwantmyname.com/?domain=somedomain is a good example interface for domain name search.

You would basically set up a user interface with 25 rows, which contain the name of each of your databases. The search term would then be entered at the top and this would fire off AJAX requests to 25 or less adapter scripts that each search one of the 25 databases. As results are returned they appear as single entries or lists in each of the 25 rows.

Answer 2

You implement a form of microservice architecture - each of the 25 sites has a 'front end' (to them) that is a back end to your website. You make requests to each of these which make the appropriate request to the sites.

Its up to you how to aggregate the results you get back, whether to take them as they come, or to hold them and process them into a single form. The main point is that your web server makes requests to services that represent each of the 25 customer sites.

I did this for a credit checking system - a single user form showing all a person's credit history was aggregated with requests to several systems containing bank data, credit card, tenancy, electoral, fraud etc. We hit each one requesting details for Nick and combine the results afterwards. Each system, you'll note, was different in the data held, the format, and the way the request could be made. Some would return the data you asked for (as it cost money per request), others would be queried twice - once for a search hit and then again for details, others were regularly imported into a locally-held DB and searched on.

The point here is that the webserver doesn't care how the data comes back, only that it can make a common request on each microservice, each microservice only cares about how it gets data from its particular back-end system.