By suggestion in comments, I'll try to rephrase my question to better reflect my problem:

I have to present users (few at first, as many as possible later) with a website. There they will be able to browse various data. The data will be served from a local "cache" db. Users should be always presented with data as fresh as possible.

The source for most of the data will be external services with possible limits, long response times etc. The process of processing data from those sources to the schema of local db is also not straightforward (conversion, calculations, decoupling, recreating relations etc.)

Is putting all that data mining and processing code ok (and doing all that work in the time between users click a link and receive data), or is my second idea - another, constantly running data-miner application - better? Or is there another option?

Old text:

I'm currently developing a website that will present both its own data (user profiles, their submited content) as well as data obtained from various external sources (e.g. via webservice calls).

Varied sources will have varied limits of allowed calls, will respond with varied speed etc.

No, the data user is presented with should be, by default, as fresh as possible with an option to request refresh (unless refreshed very recently).

I started development in ASP.NET (for better or for worse), with some templates and some customization, and quite a lot of coding I ended up with a workable proof of concept. But then I started to think about large-scale operations.

Displaying cached data is quick and easy, I didn't mess that up. Refreshing though is a problem.

a) Requesting data for one user is fast, but it will eat through allowed api calls quickly. b) Requesting data for multiple users (for apis that allow that) saves potentialy great amount of api calls, but takes linearly longer (not 1:1, but still, it takes time to request, receive and parse tens of times more data than single request).

There also comes the problem of exceeding the limits. Usually that would force the app to either cancel refresh, or wait few seconds for new allowance. But I don't really see the user happily waiting for several seconds for the page to load.

I am currently concidering gutting the data-gathering part of the website and moving it into a single managing application, sharing a db with the website. The website would provide info which users need their data to be refreshed, the app would load-balance the external services to use them efficiently (not limited to "when a user opens a website"), and keep the data as fresh as possible - also possibly keeping part of the allowance for forced refreshes.

Now, the main problem I see with this is cost of hosting not just website, but also a background running .NET application. But let's assume that the cost is not that relevant. What I actually fear is that I overcomplicate the solution. I might just put more of the website in Ajax and just wait for the request to complete in the background (when calls become available).

So, does anyone has any simillar experience? I'm especially interested in the scenario where it is all scaled up, that is we can assume that there will always be deficit of available calls.

PS I also worry about keeping website (which can be triggered independently by many users) from trying to refresh the same records twice etc. - I guess I should put some additional safeguards, lock or at least mark the records undergoing refresh etc. - a lot of code just to solve the problem that wouldn't exist with a single-backend....

  • 2
    Any answer possible for this question is going to depend too heavily on which services you are calling. The limitations placed by one service provider will be nothing like any other. For example, Google Maps API says that a business may make up to 20K calls per day to the latlng API (from the business' server), but if you shift the API calls to the browser, each unique browser can make up to 2k calls per day. Yahoo will rate limit you when you exceed their limit. Mapquest just cuts you off until tomorrow. Mar 30, 2014 at 15:41
  • Well, some api's require to pass a secret key that they use to monitor call usage - so I cannot push the calls to the user sadly. Also, my problem is not really depending on the fact that I have limited api usage, but in general, lots and lots of data mining and manipulation in what should be basically a website.
    – Gerino
    Mar 30, 2014 at 22:32
  • 1
    My point is that there are too many variations to be able to answer your question as is. Try narrowing the scope of your question. That will help with the possibility of an answer. Mar 30, 2014 at 22:43
  • I've done this. Pretty soon you really need a separate task, and most likely some kind of scheduling and communication framework. At this point, which tech you are using really affects any advice.
    – david.pfx
    Mar 31, 2014 at 10:52

1 Answer 1


If you're making calls to a 3rd party service and that service's response is not dependant on the user's request then yes - consider making a data miner gateway between you and the service.

The key points to decide are whether the 'cost' of calling the service is prohibitive, if you are calling it repeatedly to fetch the same data then there's no reason why you need to call it on every request (eg a stock price say, where users get data every 15 minutes, you can simply call it once every 15 minutes to refresh the price). If however, you do need to update it based on the user, then its going to weigh heavily in favour of calling every request and the data miner will not work well.

I have done similar things - usually for data that is received independently of the user requests - eg GPS signals for a vehicle, the data comes in when it likes and it got stored in a dB that the user would read from when the user needed to know where the vehicle was.

Waiting for multiple users to request the same data is a poor choice though - make the call for the first user and cache the results, if another user requests that data within a time limit, respond from the cache instead. Where the cache is not filled with the right data, let the cache make the request to the 3rd party to fill itself while blocking the user request, then the newly-filled cache will be the only interface for all user requests - that kind of architecture makes life easier.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.