2

I'm currently working on a service that is supposed to aggregate data from a number of APIs, unify the data, and offer it through another API to the users. I had a couple of ideas for solving this, but I'm not confident my best solution is the best possible solution.

Solutions I considered:

  • Let the browser poll our API through javascript, which in turn polls the third party APIs. This introduces quite a bit of lag between the request and the user seeing the data, but doesn't waste resources on users that rarely log in.
  • Let the browser poll the third party APIs and afterwards send the data to our API. This is potentially insecure, and requires me to provide extra API endpoints and a lot more data cleanup.
  • The option I currently consider the most optimal is letting a service on the server (an actual executable) periodically poll the third party APIs. I would, for example, poll some endpoints every 30 minutes, and others maybe once a day. This is the safest option, and introduces little lag to user requests, and prevents users from reaching the request limit of the API. But data will only be up-to-date right after polling.

Are there any alternatives? If not, is my current optimal solution good enough?

2

Another option is not to poll at all, but to implement a caching solution where, when a request comes in to your API, you use a cached value if you have one that's recent enough, otherwise you update your cache with a call to the external API.

1

If you have control over any of the apis, and for your own api if you are expecting the browser to display "live" updates, you may want to consider implementing "long polling", also known as "comet". In long polling, whena request is made, the server checks to see if anything has changed since the client last polled (it can be helpful if the client includes a time stamp from the response to its last poll for working this out) and if so returns the changed data immediately. Otherwise, it waits until data does change before returning. Therefore, the client only needs to make a single request and wait for the data to come back, rather than n having to continuously poll. This reduces load on the server and makes clients respond faster.

  • 1
    Unfortunately, the APIs are all third party, and while some may implement "comet", it certainly isn't available in all APIs. They don't even all provide HEAD, which I could use to compare timestamps without requesting all data. – Simon Verbeke Dec 18 '14 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.