Using external APIs to build a dashboard

Question

I have a website that is based around some external APIs related to a certain theme. It combines data of responses from multiple APIs before returning to the frontend.

With that said, my server side does not have a database. Instead, it requests data from the external APIs, and returns things to the frontend, without persisting anything. It does not need to persist since that would create an overhead from syncrhonizing data between my database and the external APIs.

But, I have a new requisite now that envolves creating a dashboard. This dashboard depends on a comparison of data between the entity being requested, and all the other ones that exists.

For example: Suppose entity A expends $ 10 per month. A dashboard field would inform how far away entity A is from the average money spent by all the other entities. I would have to read the expenses from all other entites before calculating the average.

I cant build this dashboard with transient data, like I have been doing. I will have to persist data from all entities in order to supply this dashboard.

What should be my approach to it?

1. Should I create a feature on my server that periodically reads and persists all the data from the external APIs? I would then also need to calculate and persist the metrics that would be shown on the dashboard for each entity.
2. Should I download that data to memory on my server and persist only the calculated results? For example I would download all the expenses from all entities, calculate the average money spent and persist it as a metric on the database.

Answer 1

As you found out, this new requirement can not be satisfied with your current architecture (no local database) and the external API which would be the right place where statistics should be computed and be made available.

As developers, our normal reaction is to ask ourselves "how do I need to change my system to fulfill these requirements?"

However, sometimes it is necessary to think outside the box. The effort to implement a solution within your system is considerable, and depending on your relationship with the provider of the external API and the number of entities querying all entities regularly might not be acceptable.

So to support management in finding a proper solution, you might want to estimate your effort and provide it to management along with an assessment of the the possible risks and a suggestion that working with the API provider to get a statistics endpoint might result in a cleaner solution and might also be cheaper (this highly depends on your relationship with the provider.)

If you stay on the current route, then the choice between your alternatives depends on the needs of your application, it's basically impossible to tell which is better without knowing the details.