I'm currently working on a service that is supposed to aggregate data from a number of APIs, unify the data, and offer it through another API to the users. I had a couple of ideas for solving this, but I'm not confident my best solution is the best possible solution.
Solutions I considered:
- Let the browser poll the third party APIs and afterwards send the data to our API. This is potentially insecure, and requires me to provide extra API endpoints and a lot more data cleanup.
- The option I currently consider the most optimal is letting a service on the server (an actual executable) periodically poll the third party APIs. I would, for example, poll some endpoints every 30 minutes, and others maybe once a day. This is the safest option, and introduces little lag to user requests, and prevents users from reaching the request limit of the API. But data will only be up-to-date right after polling.
Are there any alternatives? If not, is my current optimal solution good enough?