Alternatives on how to deal with data fetched from external API

Question

We are planning to build a web application which will consume data from an internal REST API that centralizes all of our "core" business data. For matter of demonstration, let's say this API exposes data for a very large Products database. So an example of our web application using the API would be:

A View that lists all products and the user can add products to his "wish list". (Very silly example)

Now, the difficulty is: each time we would have to know about a user products, we would need to send a request to the API to get the products names and other data, since we would only save its Id internally. This is okay in this simple example, but there will be cases much more complex than this (Reports, for instance) and it just seems very wrong to always fetch data from the API. So we came up with several approaches:

1 - Fetch the products from the API on views that create records and save then internally in our database. The other areas that uses the products, would query it from our Database not from the API.

2 - Some sort of caching strategy. But where? Who would be the responsible for the caching? The API or the client? If so, what strategies would better suit this case?

Case 1 sounds like an plausible solution but it would create a problem of synchronization. We would need to check often to see if changes were made (its name, for instance).

Case 2 sounds like a good one but I personally never have seen a similar architecture before nor had implemented one.

Another thing that came up was, If we only save the Id's in our local db, then when we need to "build" the view with all the data, we would need to query our local db getting all the ids of the user products, send a request to the API to get the products. The problem? The API doesn't know about the relationship between user -> products so we would need to query all products and somehow merge the data in our side.

Any kind of advice, shared experience or basically any input would be much appreciated.

PS1: Both the API and the Web app are written in C#.

PS2: When I said "Internal API" in the question, I meant that it is an API designed by our team. The external in the title means that the API will be separated from our web app, acting like an external resource.

Answer 1

If I'm following correctly you are considering using an API that only provides a dumb interface into your data, ie you can't query through the API you can only request all the data in one go.

So if you want to look up the Product for one user you have to fetch all, lets say, 20 million products and on the client side filter through them for the ones that belong to the user and then only display them.

If that is the case then yeah that is going to suck.

The best solution is to ask the API team to change the API so it can be queried so you can ask it for specific resources not just all or nothing.

If that isn't possible then try and get them to put a cache in front of the API call so at least it is fast.

If you can't do that then another alternative is to use something like memcache on your end to save each dump of the data. You can control when you invalidate this cache (say every minute or so).

Ultimately though if the data API isn't querable you are going to have a hard time with what ever solution you come up with, so if you can press that team to make the API better

Answer 2

If you want to ease load on your servers where the webservices providing REST endpoints live, there is no other way than to introduce caching to these servers.

You could specify a constraint that a client would have to introduce cache in his application so he does not hit your services that often, but how are you going to guarantee that is really going to happen, even when the client is still the same company.

If you want something reliably done, introduce it in your own system. Never expect others to do the heavy lifting.

---

By the way the whole point of encapsulating database behind web services is to introduce middlewares to intercept the requests, which may then go to a database but can also be forwarded to systems like Redis or Elastic.

If a client calls your API to retrieve data A from you but then based on the value A has to manually query the database himself, the middle layer makes little sense.

Everything should go through it.

---

On the other hand, if your web service is stable and fast, providing responses quickly, and the client still feels like the calls to api are the bottleneck of the application, then it is the clients responsibility to make his application faster, not yours.

All in all hitting web services to retrieve data is not considered bad, unless you are limited by the amount of requests you may perform during a day.

Do not overcomplicate stuff.

Do you feel like the responses from web services could be faster? Introduce caching to the system consuming the web service requests.

Are the responses from web services fast enough but client application is still slow because it needs to constantly call the web services? Introduce caching to the client app.

Answer 3

You said:

Now, the difficulty is: each time we would have to know about a user products, we would need to send a request to the API to get the products names and other data, since we would only save its Id internally. This is okay in this simple example, but there will be cases much more complex than this (Reports, for instance) and it just seems very wrong to always fetch data from the API

I'm not sure it is wrong to always fetch data from the API, initially at least. You're talking about caching, but as soon as you introduce caching, you have to worry about invalidating and updating that cache, and that's always the major problem.

So stepping back for a moment, do you know you're going to have a performance problem ? Given the complexities of introducing reliable caching, I would perhaps get a solution up and running, and then determine where your bottlenecks are before introducing performance enhancements.